I don't want to sound like a cop-out, but this is one of those "see a lawyer" questions. Or a tax professional.
I think your question is irrelevant. When your father died, the ownership of the house passes to his estate, which is a separate legal entity. The estate needs to file its own tax return for any income received by the estate, such as the sale of the home, or pension payments and business income received after the date of death. The estate passes the proceeds to the heirs. Now, I'm pretty sure the estate gets the house with a stepped-up basis equal to the FMV on the date your father died, which means neither the estate nor the heirs will owe capital gains tax after the house is sold and the proceeds passed to the heirs, unless you sell at a profit relative to the date your father died.
In other words, you don't need to do any calculations about what your father's cost basis was, because the estate gets a stepped-up basis.
I don't think the fact that your father never recorded the deed, changes the fact that he owned the home and it passed to his estate.
