Roth conversions themselves are not subject to penalty, only income tax. One could induce a tax underpayment penalty by doing a Roth conversion and not having paid an insufficient amount of tax withholding from all sources plus estimated tax payments, but I'll assume that you will make an estimated tax payment to avoid that.
Not sure why you would be are asking about opening a rollover IRA. Are you mistakenly referring to a 401(k) or similar qualified retirement plan as an IRA? If the account is a 401(k) or similar account, not a traditional IRA, a taxable rollover directly to a Roth IRA is permissible under the law. Are you considering doing the conversion with a distribution from a traditional IRA at one custodian and depositing it as a Roth conversion contribution at a different custodian? Some IRA custodians only permit in-house Roth conversions. Simplest is an in-house Roth conversion (traditional account and Roth IRA at the same custodian).
With regard to figuring the amount using TurboTax, enter incremental amounts of Roth conversion, say, $1,000 at a time, and compare that to the resulting change in federal and state tax liability. If the additional income is being taxed at a marginal federal tax rate of 12% you'll see your federal income tax liability increase by $120 for each $1,000 increase in income. Keep in mind that taxable Roth conversions will increase your AGI which can have side effects on other things that depend on AGI, for example, the taxability of Social Security income.
