from what I understand, you owned a home that you permitted your brother to live in. You did not charge him rent.
This is then a 2nd home of yours.
Determinging capital gains has nothing to do with the mortgage. So paying down the mortgage on your princple home (or the home you are about to sell) won't change anything.
The capital gains on the house you are selling is:
1) Selling price - sales expenses (the commission is normally the biggest expense)
2) what you originally paid for the home PLUS any improvements
1) - 2) is the gain
