Form 8889, Part II, line 15, specifically asks for the amount of Qualified Medical Expenses paid using HSA distributions. If your total distributions are greater than the medical expenses plus excess contributions withdrawn before the tax filing deadline, you owe tax and a penalty on that amount, assumed to be withdrawals for non-medical expenses.
