"Proceeds" is not the net amount of money that was distributed to you. It's the amount you sold the house for. You can subtract selling expenses and real estate tax paid at closing. TurboTax asks about those things. But you cannot subtract paying off the mortgage loan. You don't take the mortgage payoff into consideration in calculating the proceeds of the sale.
You pay tax on the gain, not on the "distribution of money." The gain is the amount you sold the house for minus the basis. As tagteam said, the basis is the fair market value on the date of your wife's father's death. And again, you cannot subtract the mortgage payoff from the selling price. You do not take the mortgage payoff into consideration at all.
