They would only take your refund if an offset was placed. In general, they only do this when you are in default of the loan. If you were never in default and are on a payment schedule (even if your payment is zero), then they would not take your refund. But, if you were previously in default, then its possible you may still have an offset.
You can inquire as to whether there has been an offset placed on your account by calling the Department of Treasury Offset Program at 1-800-304-3107.