It is not clear what you actually did. If your return was rejected you should not have clicked on anything that said "amend." Amending is a special process that you should only use if your return was filed and fully processed with a mistake or missing information. An amended return requires the use of a Form 1040X which must be printed, signed and mailed.
Some people mistakenly use the term "amend" when all they mean is that they made a change to the return before filing it, so it becomes confusing when we try to understand what you did.
What SHOULD have happened when your return was rejected, is you should have printed, signed and mailed the return claiming the children. Then the IRS would contact both parties to sort out who could legally claim them as dependents.
If you can explain more clearly what happened with your tax return we can give better advice.
Here's some info, you need to be aware of:
The custodial parent has first priority on claiming the children on his/her taxes; regardless of the amount of support provided by the non-custodial parent. The IRS goes by physical custody, not legal custody. The non-custodial parent can only claim the child as a dependent if the custodial parent gives permission (on form 8332) or if it's spelled out in a pre 2009 divorce decree. (without conditions - usually the payment of child support). Even if a divorce decree, dated after 2008, gives the non-custodial parent the right to claim the child, he/she must still get form 8332 from the custodial parent. A properly worded decree should require her to provide that form. https://www.irs.gov/pub/irs-pdf/f8332.pdf
There is a special rule in the case of divorced & separated (including never married) parents. When the non-custodial parent is claiming the child as a dependent/exemption/child tax credit; the custodial parent is still allowed to claim the same child for Earned Income Credit, Head of Household filing status, and day care credit. This "splitting of the child" is not available to parents who lived together at any time during the last 6 months of the year; then only one of you can claim the child for any tax reasons. The tax benefits may not be split in any other manner.
Note in particular that the non-custodial parent can never claim the Earned Income Credit, Head of Household filing status or the day care credit, based on that child, even when the custodial parent has released the exemption to him.
So, it's good idea to let the other parent know that you will be claiming those items, as many first time divorced parents are not aware of this rule and may try to claim those items, which will cause the IRS to send out letters.