Short answer, you can't deduct any points, and they don't add to basis.
Long answer: You could have deducted the points on the purchase mortgage in the year you purchased the home, or you spread the points out over the life of the loan, depending on the circumstances. Then, with each refinance, you either could have (a) deducted the remaining points from the prior loan, and then deducted the points from the new loan over the life of the loan, or (b) added the points from the prior loan to the points from the new loan, and restart your deduction over the life of the loan, again depending on the circumstances. (Deducted over the life of the loan means that, if this was a 30 year or 360 month loan, you deduct 1/360th of the points for each monthly mortgage payment you made in the tax year.) Then finally, when selling the home and paying off the mortgage, you can deduct any remaining points that are left, based on 26 years of payments and calculations.
The amount of points you can deduct in 2023 is the amount of point that you could deduct if you had kept those records for the past 26 years of payments, refinance points, and so on. You can't deduct all the points you never deducted, only the points you could normally deduct now if you had been deducting them all along.
If you have reliable records and you can determine what your remaining points are, that is what you can deduct this year. Keep accurate records because, if audited, the IRS does not have to award any deduction that you can't prove with reliable records.
Also FYI, the only adjustments to basis that are allowed are items that you would have had to pay even if you bought the home with cash. That might be inspection fees required by local law (if a radon inspection is required, for example), or transfer taxes and mortgage recording fees. Fees that you only pay to obtain a mortgage (including application fees, appraisal fees, and points) are never added to basis. Points can be deducted as a form of mortgage interest using the interest deduction, not a basis adjustment.
