Advance Childcare Tax Credit Issue

I received a letter from the IRS saying that they found a discrepancy and instead of my full refund I now owe $341 because I accepted the Advance Childcare Tax Credit and didn't report it. I didn't realize I could opt out and my ex could opt in, so I was sent the advance tax credits via direct deposit and then I turned around and I sent all of that money to my ex who will claim him this year as he was technically entitled to it. I can prove I sent these payments. Is there any way to fight having to pay since I did not keep the money myself?