People recommend investment with HSA money. Does that mean I pay 20% penalty if I do that? i.e) $4000 from HSA money. will I pay $800 since it's not a medical expense?

I had $4700 with my ex company HSA account. And it was just consuming $8 or so for maintenance fee. So, I thought I could use this money to invest and recover those maintenance fees.  

So, I transferred $4000 to the investment account (they're the same company). Now, I got 1099-SA and it says $4000 as gross distribution.  Am I paying 20% penalty?  

When people say investment with HSA money, does that imply 20% penalty since investment is apparently not medical related category. 

Can I repay ?