The IRS doesn't care about any agreement you may have. They have their own rules that are not subject to the rulings of any court below a federal judge. Since federal judges don't handle divorce or custody cases, you have to go by the IRS's rules. macuser has posted below what applies to your situation, as we understand it, based on the information you provided in your post.
