What product did you use?  For example, if you used Online Deluxe prior to March 1, the fees would have been as follows:

Federal return $40

State return $40

Option to pay fees out of the Federal refund:  $40 service fee ($45 for California filers)

Plus possible sales tax.

So that would be a total $120-125 plus possible sales tax--if you used Deluxe..

Are you sure the $40 on your card is not the Federal refund after fees?   You said you "assume" it is the state, but it may not be.  If you used Deluxe, that's about the size your Federal refund should be after the fees are subtracted from the $173.   Check to see who made that deposit on your card--Federal or state.

Also, look at your state return to see what amount your state refund is supposed to be.