See this info on NY residency
https://www.hodgsonruss.com/assets/htmldocuments/Residency%205.22.2020.pdf
It is possible to be domiciled in New York even though you live most of the time someplace else. And New York is certainly allowed to send you a letter asking you to re-file your taxes or assessing additional tax, if they believe you are domiciled in New York State.
If you believe you are not domiciled in New York State, you will need to send proof and make that argument to the New York State tax department.
your domicile is your permanent and true home. You can only have one domicile at a time, no matter how many different homes you might live in. Your domicile is where you have the most ties; financial, family, legal, social, and so on. No single factor decides, it is a combination of all factors including where you live, where you work, where your doctor and lawyer and dentist are located, your church and other significant social relationships, your family, your children‘s schools, your voter and driver registration, and so on. To change domiciles, you must both establish a new domicile and take steps to actively abandon your previous domicile (such as by selling your home or abandoning your lease). It is possible to live away from your domicile for a long time, even years, without actually changing it.
If you believe you are domiciled in California, then you don’t owe any New York State tax, unless you temporarily work inside New York State during the year, or you are doing remote work for a New York State employer for your own convenience.