The tax code says that in a year for which you are required to take an RMD, the first amounts distributed are RMD. Because an Roth conversion is a special type of distribution and rollover, and an RMD is not eligible for rollover, the RMD must be satisfied before doing any conversion.
If you to the Roth conversion first, an amount that is ineligible for conversion (because it is RMD despite your desire for it not to be) is deposited into the Roth IRA and becomes an ordinary Roth IRA contribution, not a conversion. If that ordinary Roth IRA contribution exceeds the maximum that you are permitted to contribute for the year, you have an excess contribution to the Roth IRA.
Even if you were permitted to do a Roth conversion before taking the RMD, the only case I can think of where if might be beneficial to do that is if the year is the first year for which you are required to take an RMD and you wanted to delay the RMD income to the following year by taking that distribution in that following year by April 1.
