Do not worry about which spouse the distributions were made for.
It sounds like you two have only one HSA, which is in your name (since HSAs are individual trusts similar to an IRA, your spouse could have one, too).
It does not matter to your HSA if you spent money on yourself, your spouse, your dependents, or whatever.