1) Do I have to pay both NJ-1040 ($200) and NJ-1040NR ($5500)?
Yes, you have to pay the amounts due on both New Jersey tax returns. The Form NJ-1040NR is for the part of the year that you were not a New Jersey resident but worked in New Jersey. The Form NJ-1040 is for the part of the year that you lived and worked in New Jersey. Income that you earn for working in New Jersey is subject to New Jersey tax, whether you are a resident or a nonresident.
2) Why didn't TurboTax tell me to pay NJ-1040NR?
It did tell you, but not in a clear and consistent manner. There does seem to be something of a design defect in the New Jersey interview for your particular situation. Some of the screens show only the amount due on the part-year resident return (NJ-1040), and not the amount due on the part-year nonresident return (NJ-1040NR). But it did tell you that you were filing two separate New Jersey tax returns. And when you printed your New Jersey returns, right after the filing instructions there were two payment vouchers, NJ-1040-V and NJ-1040NR-V, with different payment amounts. Unfortunately there are a couple of places in the filing instructions where it mentions only the payment due on the NJ-1040, but the "What You Need to Mail" section shows the payments due on both returns, and mentions both payment vouchers. If you had reviewed the printed returns more carefully before you mailed them, you would have seen the two returns and the two payment vouchers.
I'm sorry you had this problem. Your situation is somewhat unusual, but the treatment of it in TurboTax certainly could be improved. I will bring it to the attention of the community moderators, who will pass it on to the software development staff.
