Don't fool around. If you are not supposed to claim your children for 2017 by court order, don't file an amendment now. If he did claim them, you will start an investigation, and you could end up annoying the judge overseeing your case.
However, it is important that you are aware of your rights and responsibilities when sharing custody of dependents. Only the parent where the children live more than half the nights of the year, has the automatic right to claim the children as dependents. If that is you, then for the years when your spouse is allowed to claim them by court order or agreement, you must give your ex a signed form 8332 dependent release. This allows your ex to claim the child tax credit only (and dependent exemption for 2017 and earlier). The ability to use the child to qualify you for earned income credit, dependent care credit, and head of household status always stays with the parent with custody more than half the nights of the year, and can't be waived, transferred or shared. If your ex tried to claim the children without the form, and you claimed them for EIC or HOH, then he would get a block. He can amend and do it right if you gave him the form. If you didn't give him the form, you are at fault in this case.
Also, if you did not list your children on your return, you might have missed some tax benefits. In this case you could amend, but only to claim what you are due, not to try and grab the dependent against the court order. In Turbotax, you would answer that the children lived moire than half the nights of the year (choose 7 months or more); there is a custody order; you will give the other parent form 8332. This lists your children as "non-dependent, use for HOH and EIC only" status. You or your accountant should check to make sure that you did use the children for HOH status and EIC (if you otherwise qualify) -- that would be a valid reason to amend.
But I suggest you not mess around with violating the court order. If you amend and get the children as dependents, and then your ex figures out what he did wrong and also amends, you are in a mess.
To answer your direct question, it is not too late to file an amended return for 2017 to claim an additional refund, and the IRS does not "reject" amended returns (that only applies to e-filed returns and amended returns can't be e-filed) but the IRS can certainly send a letter denying the change or starting a custody investigation.
