Any tax refunds that were kept would have been applied to the debt and reduced the total amount you owed. If you feel the bank did not credit you for the amount of your tax refunds that were held back, you would need to start by checking your account at the bank or lender, and working with them to show them proof of offsets that were collected, etc. and make sure they gave you proper credit in your account.
The agency that administers the offset program is called the Bureau of Fiscal Services or BFS. If your bank does not give you satisfaction you can call the BFS. https://www.irs.gov/taxtopics/tc203.html
Ultimately you may need to sue your bank if the BFS says they paid the bank and the bank won't credit your account.
