If you have 1099-Rs for those prior years, you should amend your returns for those years to include them (unless the IRS did it for you in a notice, and even then you may have to file to get the full benefits). Technically your distributions are income, though it appears from what you said that these were direct rollovers with no net change in income or taxes.
