My wife and I took some 401k payments this year and it effected my tax payment amount. it was a one time thing why do i have to pay back part of my stimulus?

my actual income didn't change adn we used the money to pay off our mortgage.