If you are divorced and are not liable for the taxes your Ex owes, your refunds should not be offset.
First, I would recommend contacting the Bureau of the Fiscal Service (BFS) at 1-800-304-3107 to check with them about any potential refund offsets for you.
Second, I would recommend checking your IRS Tax Account online to see if they show that you owe taxes by clicking this IRS link: https://www.irs.gov/payments/view-your-tax-account.
You can contact the IRS at 1-800-829-1040 if you find that your refund is incorrectly being offset for your Ex Spouse's debts.
If you are unable to get things resolved with the IRS directly, you can contact the Taxpayer Advocate by clicking here: https://www.irs.gov/Advocate/The-Taxpayer-Advocate-Service-Is-Your-Voice-at-the-IRS. The Taxpayer Advocate is there to help you deal with the IRS when you are unable to get things straightened out on your own.