"The Roth conversion was done in March, and there would definitely be a loss if the excess contribution was returned, so there won't be any additional tax involved, correct?"
Correct. Be aware that the gain or loss is calculated with respect to the entire Roth IRA account.
"If I understand it, doing the excess contribution step is fairly easy. I would then do a new rollover with the $50,000 and I would basically be in the same place tax wise, except now I have the RMD and Roth conversion in the correct calendar order. I don't think this will have much of a financial impact, correct? If I do this, will I have to fill out form 5329?"
Correct, no real financial impact. With the corrective distribution made from the Roth IRA, there is nothing to report on Form 5329. If the amount originally moved to the Roth IRA as an intended Roth conversion was more than your RMD, the RMD has been satisfied. The only reason you would have Form 5329 with regard to all of this is if by the due date of your 2022 tax return, including extensions, you fail to correct the amount now in the Roth IRA which must be reported as an excess contribution. If corrected, you won't have an excess contribution to report on Form 5329.
"One concern is if Vanguard will process this correctly and the correct 1099-R form will be generated. From this conversation, it sounds like the 1099-R form would be almost the some if I didn't do this, correct?"
The code-7 Form 1099-R will total the distributions from the traditional IRA. There is nothing special about that. The return of contribution must be requested as such, not a regular distribution, so that the Form 1099-R reporting this corrective distribution will have codes J and 8 if done in 2022 or codes J and P if done in 2023 before the due date of your 2022 tax return, including extensions.
