If your state allows the unemployment exclusion, the refund is based on the tax that was calculated on your state tax return, not on the tax that was withheld from your unemployment benefits. You paid tax on the unemployment when you filed your tax return, even though you did not have any tax withheld.