The amount of refunds or amount due are 99% due to proper withholding. I assume you are an employee?
Withholding tables assume that, for whatever your pay is, per week, biweekly, or monthly, it's going to be consistent all through the year. If you have pay periods that your income varies widely it's tricky to withhold properly. Also, if you worked a second part-time job, that job's withholding will often be less than the tax burden on that extra income.
If you compare your tax liability for 2019 taxes vs taxes withheld with your 2020 taxes vs taxes withheld, comparing ratios might give you the answer.
