Either the IRS or your State can garnish your refund. It is called a refund offset. A refund offset is when an IRS refund is reduced or intercepted to pay off delinquent debt, such as past-due child support, outstanding student loans, or unpaid state income tax.
You'll get an explanatory letter from the Bureau of the Fiscal Service if your refund is offset. The remainder of your refund will be processed as usual; an offset shouldn't delay it, unless of course your entire refund was applied to your debt.
More info about refund offsets, including contact info and the appeals process, is on IRS.gov (Tax Topic 203).
If the refund offset came from your State you should also receive a notification. You can contact your states Department of Revenue to get an explanation. See this FAQ with a link to each states taxation department: https://ttlc.intuit.com/replies/3300211