A return of contribution (an RMD impermissibly converted is a failed conversion and therefore constitutes an ordinary contribution to the Roth IRA) is paid to the participant, not returned to the traditional IRA. The excess is the dollar value equal to the amount of the RMD that had been impermissibly deposited into the Roth IRA. A return of the excess before the due date of the tax return would be the distribution of that dollar value plus the net attributable income calculated based on the investment performance of the entire Roth IRA. The fact that a Roth conversion was done in-kind is irrelevant. You can think of the conversion as having been done by selling the shares in the traditional IRA, conversing the cash, then in the Roth IRA repurchasing the shares at the same share price.
