According to IRS.gov, if you wish to dispute the offset, you should contact the agency that received the offset payment (loan company). Only contact the IRS if your offset payment was applied to a federal tax debt.
You can also try contacting the Taxpayer Advocate Service for assistance.