So I am getting double taxed?

No, you'll be taxed on the $20,000, and the $4,000 you had withheld on that distribution will be used to pay your taxes on the distribution.

If you are under age 59½, the amount not converted to Roth ($4,000) is also subject to an early-distribution penalty.

Roth conversions are most beneficial if you can pay the taxes with funds from another source since you end up with more in the Roth IRA to grow tax free.

Good point, I definitely don't want to incur the early-distribution penalty.