As you now realize, HSASs are not like typical savings accounts - there are rules on how you contribute money and how you remove it.
What you should have done when you had this money from your HSA in hand that you realized you should not have taken out, is to contact the HSA custodian and reported a mistaken distribution. The custodian, if they accepted your request, would have had you fill out a form and asked for a check for that amount - yes, the same amount you innocently sent to the HSA.
The difference is that now the "contribution" would be recorded as a repayment of a mistaken distribution and not as a contribution.
Please read the section under "HSA Mistaken Distributions" in the instructions for form 1099-SA. NOTE: as you see, they don't have to accept your request, so be nice (and grovel if needed). You are asking them to convert the contribution to a repayment of a mistaken distribution. This is so their records will be correct.
Good luck.