"So now they say Roth-IRA company will have to update my 1099 or send me two of those or something like that."
It's the 401(k) that will be issuing Forms 1099-R. They should treat as a corrective distribution the $10k excess and attributable earnings that was part of the rollover to the Roth IRA and report as rolled over only the part of the 401(k) that was no excess. If the attributable earnings represented a gain, that gain will be taxable.
The $10k and attributable earnings was not eligible for rollover to the Roth IRA, so that must be treated as a regular contribution to the Roth IRA, an excess contribution to the extent that it exceeds the amount that you were eligible to contribute for the year. You need to inform the Roth IRA custodian that the $10k plus attributable earnings must be treated as a regular contribution so that the Form 5498 for this account is accurate and request an explicit return of contribution to avoid excess contribution penalties each year until the excess is resolved.
You do have the option to leave the excess contribution in the Roth IRA, pay the 6% penalty, then apply the excess to the following year's Roth IRA contribution each year until the entire excess is resolved, but unless you have large investment gains in the Roth IRA since the rollover, which seems unlikely in today's environment, you would probably be better off obtaining the return of contribution. The deadline for obtaining a return of contribution is the due date of your tax return, including extensions.
