Are they looking to charge me a % of what I owe the feds? "A convenience fee of 2.49% of total amount you owe will be charged by our credit card service provider. "

Yes, by the IRS.

https://www.irs.gov/payments/pay-your-taxes-by-debit-or-credit-card

Yes, using credit card to pay government will always get extra charge. To avoid, you'll have to do direct draw from your bank account, or mailing a check to them.