My employer overpaid Federal Income Taxes on my W-2. He wants me to repay him without amending my W-2 or filing a 941X. Is this legal?

I already filed taxes and received a refund.  He claims that I need to give him my refund.  I disagree.  I believe that he needs to file a 941X and give me an amended W-2.  I refile and pay back the IRS and he receives the overpayment from the IRS.  Am I correct?