The contribution limit has nothing to do with the distributions that you did (or did not) take. You may leave funds in the account year to year without penalty.
You and your employer are only allowed to contribute a set amount to the HSA each year ($3500, by your calculation). Anything over that limit will be declared as income and have an additional 6% excise tax added to that, each year, until corrected.
Your employer can ask for their contributions back, or if you made the contributions, then you can ask for them back. I believe the cutoff is the end of the tax year.
