February Roth conversion taxable because of December rollover??

Here's what happened: On January 1st 2016, I had no money in any IRA account. I contributed $5,500 to an 2016 non-deductible IRA later in January and put it in a mutual fund. In February, I converted my entire IRA, now $5,400 to a Roth IRA. Because there were no gains, it should not be taxable. My IRA sat empty again. In December, I rolled over $10,000 from an ex-employer's 401k to my empty IRA. On 12/31/2016, my IRA balance was $10,000. Based on the IRA Distribution worksheet, it looks like I'm going to owe tax on my February conversion because there was a balance in my IRA on 12/31/2016. Am I reading it right? If so, no fair, IRS!