The IRS zeroed out the federal withholding from my annuity without telling me and now wants me to pay $197 underpayment penalty. Is this proper?

I am a federal retiree. OPM confirmed IRS action and said 50% of retirees did not notice the IRS change - so why should I have to pay a penalty?