Your refund goes down once you enter your W-2 because you need to make another entry in TurboTax. HSA information in TurboTax is entered in two places. You need to enter the second part in Deductions & Credits section under Medical, HSA and MSA contributions.

HSA distributions (money taken from an HSA account) are nontaxable, but only if the money is used to pay for qualified medical expenses. If the money is used for anything else, it's taxable. You should receive a 1099-SA form to enter. Enter the 1099-SA before you enter medical expenses so TurboTax can properly calculate your medical expense deduction.

https://ttlc.intuit.com/replies/4787864