What if I don't remember if I paid any points when I refinanced my home loan? Do I still have to claim it?

No, you don't have to, but it might be to your benefit to do so if you itemize your deductions. If you are taking the standard deduction, then you needn't worry about the points. 

You should see the points clearly noted on your loan documents.