No! Congress was very clear there were to be no requirement to return money,.
The way the stimulus REALLY works is EVERYTHING is based on your 2020 income, filing status and number of children under the age of 17. What these numbers were in 2018 and 2019 does not matter
The IRS will calculate your stimulus based on 2020 numbers; that is what is considered the ACTUAL.
The two payments were just ESTIMATES
if the ACTUAL is greater than the ESTIMATE, you will receive that amount as a credit on your 2020 tax return.
if the ESTIMATE is greater than the ACTUAL, (which appears to be your case) you are not required to return the money nor does it impact your refund. Line 30 on Form 1040 will be zero.
does that help?
