I'll start with the second half of your question. You entered the correct wages to remove: the New York wages. You want to have New Jersey tax calculate on the New Jersey line.
It's difficult to see where the NY State reject is coming from without seeing your return, and we don't have access to that in this public forum. However, there's a couple of possibilities you can check. First, see if your NJ and NY line amounts are the same (in box 16 of the W2). They might not be, because while in general New York income and Box 1 for Federal income are usually the same, that is not always true for New Jersey. New Jersey will tax items that are not taxed in New York or federally. If box 16 NJ is a different number than NY, then you must prorate the tax amount to the income that is taxed in New York. For instance, if your New York-listed income is 90% of the New Jersey listed income, then you will get a credit on 90% of the tax paid to New Jersey.
The second thing you can check is the tax figure that is prepopulating on the New York return for the credit. It should be the amount of New Jersey tax withheld minus any New Jersey refund or in addition to any New Jersey tax owed to be paid. If that is what is reported in the New York tax box, then use the information in the paragraph above to prorate the income to the New York portion of income.
**Say "Thanks" by clicking the thumb icon in a post
**Mark the post that answers your question by clicking on "Mark as Best Answer"
