The term is "converting" to a ROTH not a rollover so when you ask the broker to do it they know what you are doing and put in the correct paperwork.  

As to the how much ... that is hard to say exactly since it depends on your 2016 income & the 2016 tax brackets. Best thing you can do is mock in a tax return when the programs are available later this month or next month then make the conversion by 12/31/16.  If you find you converted more than you wanted to you have until 4/17/17 to undo all or part of the conversion... that is called a recharacterization. 

Tax tables are on page 44  ...  https://www.irs.gov/pub/irs-pdf/p15.pdf

Basically the answer is the $10,000 that will get you to the next tax bracket. But the great American tax system  is so convoluted, you can't always go by that the simple answer. Critter#2 is right, you need to do mock returns. TurboTax download software is better for that than online.
You could try this tool, which is available now: <a rel="nofollow" target="_blank" href="https://turbotax.intuit.com/tax-tools/calculators/taxcaster/?s=1">https://turbotax.intuit.com/tax-to...>

If your estimate is wrong and you go over into the next bracket only the overage will be taxed at the higher rate. i.e. if your $10,000 conversion puts you $500 into the next bracket you will pay the higher rate on $500 and your old lower rate on the rest of your income.
In which case it is not worth bothering to recharacterize.