"Often worked from the NJ office" is not the same as being assigned to ("tagged") to the NJ office. It sounds like you are still assigned to the NY office, but mostly work from home. I'm of the opinion that makes you subject to NY's remote worker rules ("convenience of the employer rules").
So, yes there's some risk of coming to the attention of the NY tax people.
