Your Royalty Expenses would be considered Business Expenses for your Publishing Business reported on Schedule C.
You could group them into an expense category called 'Royalty Expense' or break them down further if you desire.
Click the link for more detailed info on Reporting Business Income and Expenses.
[Edited 2/6/2020 7:44 pm]
**Say "Thanks" by clicking the thumb icon in a post
**Mark the post that answers your question by clicking on "Mark as Best Answer"