A "conversion" only applies to moving money from one qualified plan into a Roth IRA. You can transfer money from a 401(k), traditional IRA, and most other qualified pension plans into a Roth IRA and then you have to pay the income tax on the amount converted.
You can't "convert" money from a brokerage account that is not a qualified retirement plan. The only way to put that money into a Roth is by making a contribution, and the limits there are $6000, or $7000 if over age 50, but you must have taxable compensation earned from working.
The RMD has nothing to do with it. An RMD is a distribution -- money comes out of the qualified plan and goes into your pocket to be spent. A conversion is a direct movement of the money from one qualified plan into another. You should always do this by direct transfer if the plan trustees will do that (most will and all should do direct transfers). With a direct transfer you never touch the money and there is no backup withholding. If you do the conversion by rollover, you must put the entire amount to be converted into the destination account, even if there is withholding, you have to make up the difference.
For example, you want to convert $100,000. You take an $80,000 check with $20,000 withholding. You must deposit the entire $100,000 in the destination Roth IRA, by making up the money from somewhere else. If you only deposit $80,000, then you have an $80,000 conversion and a $20,000 distribution (withdrawal). If you do a direct transfer from the source to the destination account, you avoid the withholding and the entire amount gets transferred.
There is no particular limit to the amount you convert as long as you can afford to pay the income tax. Beware that since converting raises your income, it will make more of your social security taxable and push you into higher tax brackets. The effect is complicated and you would need a tax adviser to help you figure out whether you want to convert your traditional IRA balance all at once or space it out over a few years.
