The HSA administrator has no way to know how much you are eligible to contribute (other than than the fact that no one is eligible to contribute more for a year than the limit for family HSA coverage plus catch-up). The only way that your HSA knows that you have an excess contribution is for you to tell them that you have an excess contribution by submitting a request that the excess contribution be returned.
In box 3 of the Form 1099-SA, a regular distribution is indicated with code 1 and a return of excess contribution is indicated with code 2. Form 1099-SA makes no indication as to whether the amount distributed was used for medical expenses or not. It's your responsibility to file an accurate From 8889 indicating the amount that was spent on medical expenses. If you make regular distributions and a return of contribution in 2017, you'll receive two Forms 1099-SA, one with code 1 and another with code 2.