From your description, it appears that the Form 1099-R correctly reports that you rolled over to the Roth IRA an amount equal to the amount in box 5 plus the amount in box 2a, with the remainder of the gross amount going to the traditional IRA. Nothing particularly wrong with that, it just means that you have some amount of taxable rollover. It's doesn't mean that any of the after-tax money ended up going to the traditional IRA. The box 5 amount becomes basis in nontaxable conversions in your Roth IRAs and the box 2a amount becomes basis in taxable conversions. In your split Forms 1099-R, the one for you rollover to the Roth IRA will have the sum of boxes 1a and 5 in box 1. The Form 1099-R for the portion rolled over to the traditional IRA will have the remainder of the original box 1 amount in box 1 and zeros in boxes 2a and 5.
