Mortgage Points Deduction After 2nd Refinance

I refinanced at the beginning of 2020 to a 20 year loan and paid 0.75 points ($2,707.50) to lower the rate. Now I know that I have to deduct that over the life of the loan ($2707.50 / 20 years = $135.37 per year), but now I am refinancing again in 2021 since the rates have dropped even more. I am now going to a 15 year loan and will have the same payment that I have as I am paying biweekly payments now. My question is how do I go about deducting the the other 19 years worth of points from the 20 year loan? Do I deduct it all on next years taxes since that loan is getting paid off in 2021? Or is it more complicated than that?