Most of the time, you owe state income tax in the state where you live on all your worldwide income, unless you live in a state that has no state income tax. Then, there are six or seven states that will tax you if you work remotely for a company headquartered in that state even if you do not actually live in that state.
If your state withholding does not match your state tax obligations, you would need to file to get a refund from the state that incorrectly withheld taxes and you would need to file to pay tax in the state where you actually owe tax.
It’s certainly impossible to agree with your original statement without knowing a lot more about where you live and where you work and where your employer is headquartered.
