I'm afraid there is no such thing as a nontaxable rollover from a regular 401k to a Roth IRA. The amount is always taxable because the money in your 401k would have been taxed when you took distributions. Your Roth distributions, on the other hand, will not be taxable when you take a distribution, so you have to pay the tax on the rollover amount now. You do need to say Yes to the Roth question, and then you will be taxed, which is correct. (I'm assuming this was not a Roth 401k)
You might want to consider recharacterizing oyur Rollover to the Roth before this deadline under the new tax laws.
From the IRS: https://www.irs.gov/retirement-plans/ira-faqs-recharacterization-of-ira-contributions
"Can I recharacterize a rollover or conversion to a Roth IRA?
Effective January 1, 2018, pursuant to the Tax Cuts and Jobs Act (Pub. L. No. 115-97), a conversion from a traditional IRA, SEP or SIMPLE to a Roth IRA cannot be recharacterized. The new law also prohibits recharacterizing amounts rolled over to a Roth IRA from other retirement plans, such as 401(k) or 403(b) plans.
How does the effective date apply to a Roth IRA conversion made in 2017?
A Roth IRA conversion made in 2017 may be recharacterized as a contribution to a traditional IRA if the recharacterization is made by October 15, 2018. A Roth IRA conversion made on or after January 1, 2018, cannot be recharacterized. For details, see “Recharacterizations” in Publication 590-A, Contributions to Individual Retirement Arrangements (IRAs)."
