It is quite possible that your refund reduced by that much. The tax on unemployment is not a flat tax. It is based on your individual tax rate. If you did not have any amount withheld from your unemployment, you will have to pay the amount that should have been withheld.
Also, credits could have changed, such as the Earned Income Tax Credit, or the Child Tax Credit.
Unemployment compensation is taxable, and is added to your other (Joint) income to determine your taxable income. How much was the total income for 2020 for both of you?
It is quite possible that your refund reduced by that much. The tax on unemployment is not a flat tax. It is based on your individual tax rate. If you did not have any amount withheld from your unemployment, you will have to pay the amount that should have been withheld.
Also, credits could have changed, such as the Earned Income Tax Credit, or the Child Tax Credit.