If your situation is not the case below, then the points should be amortized over the life of the loan, and something got skipped over during the mortgage interest interview. Go back and verify your entries, unless -
If you refinanced with the same lender, any undeducted points left over from the first mortgage will be deducted over the life of your new loan. But if you refinanced with a new lender, any leftover undeducted points can be deducted in the year of the refinance.
Can I deduct mortgage points?