Talk to your employer to find out what's going on.
What it appears to be is that, despite the NJ office, you work for a NY based company and NY taxes the telecommuting employees of NY companies. You will have to file a NY non resident return. The NY tax people put heavy pressure on NY employers to withhold NY tax, in such cases.
If you work outside the state as a job requirement, you are only subject to New York State income tax on the days you work in New York. But if you work outside New York for your own convenience (as most telecommuter do), you are subject to New York State income tax on all your income. Pennsylvania, Nebraska, Delaware and Arkansas have the same rule.
Here's a link to New York's memorandum on its "convenience of the employer" tax doctrine regarding non-resident telecommuters: https://www.tax.ny.gov/pdf/memos/income/m06_5i.pdf
But, maybe not. See :https://ttlc.intuit.com/community/state-taxes/discussion/re-i-work-for-a-ny-company-remotely-from-nc... The Poster, who appears to be very knowledgeable, says being subject to NY tax requires at least 1 day in NY. You may be able to claim a full refund.
