Your annuity is not a wage.
But each withholding (say, 15%) is calculated as if is your only income. When you add them together, you can end up in higher tax bracket (say, 25%).
https://apps.irs.gov/app/withholdingcalculator/
https://www.irs.gov/pub/irs-pdf/fw4p.pdf
