I received the child tax monthly installments that the government was getting out for two kids my kids father now wants to claim 1 of the kids will that hold up my return

I have two children that I received the advanced tax payments for he only wants to claim one but I am afraid that that will hold up my refund and I will have to pay the portion for the one kid back I’m not sure what to do don’t want to do anything to hold up my return definitely not trying to go against the IRS either heard whoever got the payments must file for both of them