Depends on how the payee of the payment to the IRA custodian for you benefit was worded. A proper direct rollover, indicated by the code H Form 1099-R, must identify the payee actual account that is to receive the rollover, or at least include "Roth IRA" in the wording but even better if it includes the account number of the Roth IRA. However, a huge number of plan administrators fail in this regard, making the payee simply be the new IRA custodian for your benefit, allowing the IRA custodian to deposit the funds into any type of account for your benefit, including a non-retirement account. It then falls on you to make sure that the rollover was completed properly.
Had you recognized the problem quickly, the fix would have been to explain to the IRA custodian that the deposit to the traditional IRA needed to be distributed as a return of contribution since the traditional IRA was not eligible to receive the rollover, then within 60 days of the distribution from the Roth 401(k) complete an indirect rollover to a Roth IRA. The correction now is much the same, but complicated by the fact that it is now past the extended due date of your 2019 tax return and well beyond 60 days from the date of the distribution from the 401(k).
This can't be the first time that this IRA custodian has seen a situation like this, so perhaps they can tell you the procedure that they followed in the past to correct this type of error. This will require someone in their back office to address this; front-line reps will be clueless.
