If the home ceased to be your main home prior to
2017, you would have included Form 5405 in your tax return for the year it
was disposed of/sold, etc. and you would have repaid your balance on that tax
return or in some cases, the balance would have been waived. It should
not be showing an outstanding balance is due, and Form 5405 is not required.
The reject message you are receiving is because the IRS has not
properly updated its records and believes (based on their info) that
Form 5405 with a payment is due for 2017.
You can look up your account at this
link: https://www.irs.gov/individuals/first-time-homebuyer-credit-account-look-up
To resolve this, you will need to paper
file your return this year. Attach a copy of your return
for the year the property was sold, to show that you properly filed in that
year. Additionally, you will need to contact the IRS directly to
have them correct their records so you don't encounter this same issue next
year.