If the amount of tax you owe when you file your return is more than $1000, you could be subject to a late payment penalty. Quarterly estimated payments for a large gain during the 4th quarter of the year are due by January 15, or by January 31 if you file your completed tax return at that time.
To avoid an underpayment or late payment penalty, you can make an estimated payment at www.irs.gov/payments. Be sure to specify that this is a 2017 estimated payment.
To avoid the penalty, you must either
1. owe less than $1000 when you file, or
2. pay into the system (via payments and withholdings) at least 90% of this year's total tax bill, or
3. pay into the system (via payments and withholdings) an amount equal to at least 110% of last year's tax bill.
Calculating the gain will be complicated. How and when did your wife become an owner of the property with her brother? Has it always been a rental, and have your wife and her brother been taking depreciation?
Now, depreciation. When using the property as a rental, he was entitled to take depreciation (wear and tear) as an expense. You must pay back (recapture) any depreciation that he took or **could have taken** even if he didn't take it. Rental property is depreciated over 27.5 years. The basis for depreciation is the purchase price or the fair market value when it was placed in service, whichever is lower.
For purposes of this answer I will assume that while he lived there, he rented 50% and the other 50% was his personal living area.
$158K divided by 27.5 years is $5745 per year. So for 8 years he was entitled to take $2872 of depreciation and for the last 2 years he was entitled to take $5745 of depreciation. That's a total depreciation of $34,471. That reduces your cost basis in the property down to $123,529.
Then you can increase your cost basis by any permanent improvements that were made (new carpets, furnace, roof, etc.) I will assume zero for now. You can also increase your basis by some of your closing costs from 1999 that were not deductible at the time, like inspections, mortgage application fee, appraisal fee and attorney fees. I will assume $2000, so your cost basis is $125,529.
At the time her brother died, your wife inherited his half of the house, and she inherited a stepped up cost basis. Her cost basis on his half is now equal to half the fair market value on the date he died. Since the house was sold less than 4 months after he died, I (and the IRS) will assume that the sales price was the fair market value. (You said you sold at a loss, but that will be hard to prove. Fair market value is what a reasonable buyer will pay a reasonable seller. If you got less than you wanted because you were in a hurry or because the property is in poor condition, that's still the market setting the value.
So, fair market value of $250,000. That means your wife's cost basis is now half the old basis ($61,764) plus half the new basis ($125,000) = $186,764.
Then you sell for $250,000. You can subtract your selling expenses like attorney, inspections that the seller pays for, and real estate commission. So let's assume a net sales price of $230,000. That means you have a gain of 230000-186764 = $43,236.
Now, $17,235 of that gain is recapture of your wife's half of depreciation. That's taxed at a flat 25%. The rest of the gain ($26,001) is long term capital gains that is taxed at 15%. The total tax owed would be $8208.
This will vary depending on your exact numbers of course.
You will need to be diligent about documenting your cost basis and all the adjustments. (Closing costs from 1999, cost of any improvements, etc.) You may want to review your brother in laws tax returns to check his depreciation figures. If he was not claiming depreciation, that may be fixable on his final 2017 tax return, but it requires the involvement of an accountant. You may also want help from an accountant this year depending on your comfort level with all these adjustments and calculations.
Or you can contact your own independent tax expert.