When you separate from service, you are allowed to withdraw funds, or roll them over to an IRA or another qualified plan.
Since the new plan will apparently not accept a rollover containing after-tax money, I would start by asking the old 401K to rollover the after-tax portion to a Roth IRA, either with them or with another broker. Then the new plan should accept the remaining funds.
You also don't have to rollover any money to the new employer, you could rollover everything to a combination of a private IRA and a private Roth IRA. In my experience, 401k plans offer mutual funds with lower expense ratios and fees than private IRAs, but private IRAs have more investment choices. I wouldn't necessarily stick with the new 401k unless you really like the investment choices.
