As others have explained, bottom line is that if you never physically worked in CA you have no CA-source income, and thus you have no CA tax liability. (Edit: The previous sentence is true for W-2 employees, but due to a 2019 court ruling, may no longer be true for 1099-MISC sole proprietors.). If the CA FTB mistakenly thinks you have CA-source income, you'll have to set them right.
That said, it is entirely legitimate for states with an income tax to tax non-residents on income earned within their borders. Every income-taxing state does it. (The only exception is for active-duty military.) But the fact that you can take a credit for taxes paid to other jurisdictions means that double-taxation does not occur. And if you have income from a non-resident state, you may or may not have to file a tax return there. Non-resident filing requirements do differ from one state to another.
**Answers are correct to the best of my ability but do not constitute tax or legal advice.
