Do I have to amortize the points I paid refi'ng my primary mortgage. I paid the points in cash at the closing. However, it was not a refi with cash out let's say to do home improvements. Can I still deduct all the points paid on this return instead of amortizing them over the life of the loan?