If your HSA form indicates the distribution was not used for medical expenses, you will pay tax on the amount not used for medical expenses. Here is an article discussing HSA type distributions for your reference:
https://turbotax.intuit.com/tax-tips/health-care/what-is-irs-form-1099-sa-distributions-from-an-hsa-...
