Not necessarily. If the income you earned in NYC was taxed by the state at only $5,000, then you would only get a credit of $5,000. If the same income would have had taxes of $8,000 in NJ, then you would still only get a credit of $5,000 and still be responsible for the other $3,000.
If this is not what is happening in your situation, you may need to go back through and make sure your income is properly allocated and that you did YOUR NEW YORK taxes first, the the credit for NY will be transferred to NJ.
Also, only the state taxes are credited, NOT the NYC taxes.
maybe I'm not understanding how to do my taxes, but as an example, I only have income from NY lets say $100,000. My w2 says taxes paid in NY was $10,000, but when I complete the NJ return it says that I'm only credited $7,000 and I owe $3,000 to NJ. I was expecting these two state returns to cancel each other out, but for whatever reason I owe money to NJ. It's not like NJ has higher state taxes than NY...maybe they do or maybe state tax brackets are different and that's why I still owe NJ even though I don't with NY?
Your credit of taxes paid to NY on your NJ return should not be compared to your NY withholding on your W-2. It should be compared to your total tax calculated on your IT-203. Since you mentioned that you don't owe with NY, I am assuming you are getting a refund. If you got a partial refund on the $10,000 withheld from your W-2, then your tax liability was less than your withholding. You wouldn't get that amount of a credit in NJ if you got a partial refund from NY. It is based upon the amount of tax you are actually liable for in nonresident state.
Using your example, if you got a $2,000 refund in NY, your total NY tax liability would actually be $8,000. This should then be compared to the $7,000 credit on your NJ return. However, you should also note that it will never perfectly "cancel each other out" as there are differing tax rates across states. Per the State of New Jersey (page 34), The amount of each item of income taxed by the other jurisdiction cannot be more than the amount of that item of income taxed by New Jersey.