You may not contribute to an HSA unless you are covered by a qualifying HDHP and no other coverage. Medicare counts as "other coverage" and if you are covered by medicare, you are disqualified from contributing to an HSA.
If you enroll in Medicare during your initial window (from 3 months before your 65th birthday to 3 months after your 65th birthday), your coverage will begin on the first day of the month you turn 65. If your birthday happens to be the first of the month, then your coverage begins the first day of the month before your 65th birthday. If you delay your medicare, it will become effective on the first day of the month 6 months before you apply. You must enroll in Medicare unless you present proof that you have employer coverage that is as good or better than medicare.
If you make excess contributions (more than allowed based on your type of insurance coverage) the excess is subject to a 6% penalty this year and every year thereafter until it is spent, or can be treated as a qualified contribution. Your entire balance is not penalized, but the excess remains in your account and that is continually penalized until you deal with it. (Your contribution is also not tax deductible if it is considered excess.).
"Excess" is based on your coverage, not the date the contribution is made. For example, if you are covered by a single HDHP and you are over age 50, the maximum contribution is $4650 for 2022, which works out to $387 per month. Suppose Medicare starts for you on April 1, 2022. You were HSA-eligible for 3 months, so your contribution limit was $1162. It doesn't matter when you make the contribution, you could even make your contributions after you enroll in medicare. What matters is whether you contributed more than $1162. (Your exact limit will depend on how many months you were covered by an HDHP and not covered by medicare.)
I don't know if it is possible to back out of medicare if you enroll, someone else will have to advise you on that.
