My credit card was charged for the amount that I owe on my federal, but it was also removed from my refund so I double paid
Taxes owed cannot be paid from a refund, ever! If you have a tax refund, the entire refund will be paid unless an error was detected by the IRS or your State.