Or let's say I contributed $40,000 total over a few years, and then transferred one mutual fund position valued at $12,000 (that includes some earnings in that position). Would I add $12,000 to my Roth IRA basis?
The information in this example is incomplete in that it does not state the value of the Roth 401(k) prior to the distribution from the Roth 401(k). Because the distribution is not a qualified distribution, the distribution consists partly of earnings and partly of contribution basis (unless there have been no investment gains in the Roth 401(k).
If we add to the example that the value of the Roth 401(k) is, say $50,000, 20% of the Roth 401(k) is gains, so 20% of the $12,000 distribution from the Roth 401(k) is gains and 80% is contribution basis. That means that the rollover of the entire distribution to the Roth IRA adds 0.8 * $12,000 = $9,600 of contribution basis to your Roth IRAs.
If the rollover is less than the full amount of the distribution, the portion rolled over is treated as coming first from the taxable portion of the distribution. This means that the first $2,400 rolled over to the Roth IRA does not add any contribution basis to the Roth IRA and only additional amounts rolled over add contribution basis.
