No that doesn't affect when the IRS sends your refund. The fees come out of your refund after the IRS sends it. Either the IRS agent was wrong or you misunderstood.
