If past-due taxes are owed from a joint tax return that was filed in a past year, both of the people on the joint tax return are responsible for the debt. A joint tax return is not "his taxes" or your taxes. It's the tax for both of you together. The IRS can collect the tax from either of the two people on the joint return. This remains true even if you are no longer married to the person with whom you filed the joint tax return.
If you have a divorce decree that says you are not responsible for the past-due tax, the IRS is not bound by your divorce decree. Talk to your divorce lawyer about whether you can get your ex to compensate you for the tax refunds that the IRS took.
