Amortize Points refinance with same lender

I have a loan that was refinanced with a new lender in 2020, and $4000 worth of points were paid. I refinanced with the same lender in 2022 and $5000 in points were paid. How do I keep track of the $4000 worth of points that were amortized over 30 years in conjunction with the refinanced loan? My understanding is I can't deduct them all in Tax Year 2022 because it was a refinance with the same lender.