Can my ex claim my daughter without my permission

My 1 year old daughter lives with me and my parents and my parent have been claiming her on their taxes but she visits her dads house on Thursday thru Sunday every week we were never married and he doesn't pay child support he told me he can claim my daughter on his taxes without my permission and if I don't let him he's taking me to court is that true