He cannot have you audited, but if he files a paper return with the IRS also claiming your daughter, they will write you asking for proof of your meeting the qualifications to claim your daughter on your return. (This may feel like it's a audit, but it's not.)
If you have determined you are legally allowed to claim your daughter, and you did not fill out an IRS Form 8332 Release/Revocation
of Release of Claim to Exemption releasing your claim to the exemption, then you should have the information you need to defend your position. The IRS uses the tax law, not agreements such as you made with your ex, to determine your right to an exemption.
