Deductions & credits

We do not have a court order paperwork. Only thing we have is a notarized piece of paper stating that the non custodial parent claims one child for 2 years. Only reason I signed this was because he refused to sign to say I had primary physical custody even though the children live with me. My question is, do I still have to let him claim one child on taxes even though there is a notarized paper and it's not court ordered? Children both live with me. He argues constantly that he should claim one because theres a cap on how many kids you can claim, I'll be claiming 4 in total. He says he should be able to claim one because of the cap.