Are the earnings on my non taxable portion of my 401k taxable on a conversion to a roth 401k?

Yes. the earnings will be taxed.

 If you for example has a basis (after tax money) in the 401K of for example $10,000.  That basis will stay the same (unless added to) 10 years later.

So when rolled to a Roth only the basis ($10K in this case) is excluded from being taxed.