In the year that you did this, did you report a non-deductible IRA contribution using form 8606? If not, the IRS will assume the rollover was from a deductible IRA account and want the tax. Was this your only balance in a traditional IRA (combining all balances in any traditional IRA accounts at different brokers into one balance)?
