HSA Excess Calculation for Self-Employed

I am self employed and mistakenly read the out of pocket max as a maximum instead of a minimum for my HSA HDHP.  As a result, the two contributions I made were on an invalid HDHP.   I made the first contribution, 3500 in Nov, 2021 for TY 2021, and the second, 4000, in April of 2023 for TY 2022.  No investments were made on this money but I have had monthly interest accrued since my first contribution.  When I realized the mistake, I contacted Fidelity and they suggested withdrawing the full amount.  I did that on 12/05/2023. 

 

Would my excess on my 2023 return simply be the full amount I withdrew (which I know I would owe taxes on and pay a 6% penalty) or is it more complicated since the contribution errors were made in previous tax years?