Which one of you was the custodial parent? The one who the children lived with the most is the one who can claim the children. Who did they spend at least 183 nights with? If both of you claimed the same children then the IRS will look into it, and if they were claimed by someone who should not have claimed them, then that person will be subject to repaying their refund plus penalties.
Q. My kids father claimed them so did I. Will I have to pay back last years taxes?
A. One of you will. It may be you, it may be him.
The custodial parent has first priority on claiming the children on her taxes; regardless of the amount of support provided by the non-custodial parent. The IRS goes by physical custody, not legal custody. The non-custodial parent can only claim the child as a dependent if the custodial parent gives permission (on form 8332) or if it's spelled out in a pre 2009 divorce decree.
If you and the other parent live together, either one of you (but not both) may claim the child. You may decide between you which one will claim the child. Only if you can’t agree, do the IRS tie breaker rules apply, to see who has first choice. It may be worthwhile to prepare trial returns, both ways, to see which way the family comes out best. This tool may be useful: https://turbotax.intuit.com/tax-tools/calculators/taxcaster/?s=1.
A common error is when unmarried parents live together, If you and the other parent live together, only one of you can claim the child for any tax benefit. The TurboTax interview is confusing (it's designed for divorced parents, who are allowed to split the child). The second parent should not enter the child, at all.
There is a way to split the tax benefits. For future negotiations with the other parent (and maybe even for this year) the following info may be of use:
There is a special rule in the case of divorced & separated (including never married) parents. When the non-custodial parent is claiming the child as a dependent/exemption/child tax credit; the custodial parent is still allowed to claim the same child for Earned Income Credit, Head of Household filing status, and day care credit. This "splitting of the child" is not available to parents who lived together at any time during the last 6 months of the year; then only one of you can claim the child for any tax reasons. The tax benefits may not be split in any other manner.
Note in particular that the non-custodial parent can never claim the Earned Income Credit, Head of Household filing status or the day care credit, based on that child, even when the custodial parent has released the dependency to him.