Why did my refund disappear when I acknowledged that I received both stimulus payments? I had a refund of around $1200 which turned into me owing $100 to IRS

Everything I read says stimulus will not impact my refund/ return yet it has. What did I do wrong and why am I being punished?