The fact that you do sometimes work in NY puts you on the hook for paying NY tax on all your salary, for the year, unless you can find a specific exception.
If your employer is withholding NY tax, for the whole year, that's a indication that you are subject to the tax.
See this: New York Factors to apply to determine if a home office is a bona fide employer office https://www.tax.ny.gov/pdf/memos/income/m06_5i.pdf
