The amount that ended up being an excess contribution to the 401(k) was ineligible for rollover to the Roth IRA, so it constitutes an excess contribution to the Roth IRA and it was correct to remove it from the Roth IRA by a return of excess contribution. Except for the $5 of earnings accompanying the return of excess contribution, the amount returned would not be taxable as income because was after-tax money, not deferred income.
The plan also needs to correct the original Form 1099-R reporting the code 1, 2, 7 or G distribution from the 401(k) so as not to include the amount reported on the code E Form 1099-R.
