The way that credit works is unfortunately confusing. Once you understand how and why, you should be able to predict its changes.
In your instance, paying off that loan hurt you. Credit scoring looks for use of credit and good use of it. By having no debt, you're showing no credit usage. Yes, your credit history remains for some time and has some affect on your score but mostly it comes from current and the short term.
