If we were not technically residents of a state when we took a retirement withdrawal can we avoid paying the state tax on it in that state?

We had a mailing address in the state, but were not residents because we did not have a physical address, but the state is still taxing our withdrawals.