If someone else claimed your child inappropriately, and if they file first, your return will be rejected if e-filed. You would then need to file a return on paper, claiming the child as appropriate. The IRS will process your return and send you your refund, in the normal time. Shortly (up to a year) thereafter, you'll receive a letter from the IRS, stating that your child was claimed on another return. It will tell you that if you made a mistake to file an amended return and if you didn't make a mistake to do nothing. The other party will get the same letter you did. If one of you doesn't file an amended return, unclaiming the child, the next letter, from the IRS, will require you to provide proof. Be sure to reply in a timely manner.
Winner gets the tax benefits; loser gets to pay the IRS back with penalties and interest. The custodial parent almost always wins. The IRS goes by physical custody, not legal custody. The non-custodial parent can only claim the child as a dependent if the custodial parent gives permission (on form 8332) or if it's spelled out in a pre 2009 divorce decree.
You can file amended returns, for 3 years back, to claim dependents. An amended return for 2016 must be filed by 4-15-20.
References:
https://www.thebalance.com/claiming-same-dependent-audit-risk-3193030
http://taxes.about.com/od/dependents/qt/Dependents-Audits.htm
www.eitc.irs.gov/EITCCentral/f886-h-dep.pdf
