Non California Resident taxed on early 401k withdrawal

I am a non California resident but I own rental property in California.  My full time employer is not based in California so my entire 401K is money not sourced in California.  However in doing a hardship early 401k withdrawal last year.  The state looking at my federal return imposed their 2.5% tax on me.  I believe I should not have been taxed because the source of my 401k funds from my employment is not from California.