""The only difference is if you had put some money into it that was already taxed."
exactly. that is your basis.
Did you ever do that ??
