Deductions & credits

I actually paid off my rental property refinance loan early 5 years ago and have continued to amortize the original costs over the remaining 30 years.  Should I have expensed the whole remaining costs in the year I paid it off?   If so, am I now in trouble with the IRS and what should I do now?