To clarify what xmasbaby0 said, the IRS doesn't care about that "little clause" in your custody agreement, and the IRS is not bound by the custody agreement. Showing them what your custody agreement says doesn't do any good, because it has no effect on who the IRS allows to claim your son. All that matters to the IRS is what the tax law says. (They probably accepted your amended return at face value, without really investigating.) xmasbaby0 told you what the IRS does care about, which is who is the custodial parent according to the tax law, and whether the custodial parent gave the noncustodial parent a signed Form 8332.
To try to head off trouble with the IRS, you might want to talk to your lawyer and see if there is anything you can do to get the father to amend his tax return now, instead of waiting until you get that second IRS letter and have to prove that you are the custodial parent according to the tax law.
