The rules are in publication 501.
https://www.irs.gov/forms-pubs/about-publication-501
The basic concept is that the IRS does not follow state laws on dependents. The IRS does not follow state custody orders, and does not mediate custody disputes (for example, if the order says that one parent can claim a child if they are paid up on child support, the IRS does not have the time or authority to sort that out).
So, the IRS goes by a simple rule--which parent did the child live with more than half the nights of the year. That is the custodial parent for IRS purposes, no matter what the family court order says.
If the non-custodial parent wants to claim a child, they must get a signed form 8332 from the custodial parent. Without the form, the non-custodial parent can't claim the child, even if the family court says they can. The IRS will not enforce the family court order. The non-custodial parent can go to the family court and ask that they enforce the order (such as by forcing the other parent to sign the form, or punishing the other parent in some way) but the non-custodial parent can't claim the child because the IRS doesn't get involved in that way.
This answer has been verified for accuracy by an Intuit expert employee
