Your basis is what you paid for the home, plus the cost of any improvements, and minus an adjustment for business use of the home (if you used the home for a business, took home office deduction, etc.)
The amount of your mortgage or the payoff amount has nothing to do with cost basis. If you bought the home for $100,000 and sold it for $300,000, your gain was $200,000. Even if you refinanced for $300,000 and received nothing at the closing, your gain is still $200,000 (you received that gain when you borrowed the money, instead of when you sold the house).
The information and a worksheet are here.
https://www.irs.gov/forms-pubs/about-publication-523
If you owned the home and lived in it as your main home, you may qualify to exclude up to the first $250,000 of gain if single or $500,000 if married filing jointly. Those rules are also in the linked publication.
