There is no "we." An IRA is owned by a single owner. If you are the owner, and you are under ager 59-1/2, you will pay the early withdrawal penalty.
However, there is no early withdrawal penalty if you have an employer-sponsored plan like a 401(k), if you are separated from service at age 55 or higher. So, to avoid the 10% early withdrawal penalty, leave at least some of the money in your 401(k) and make your withdrawals from there until you turn 59-1/2. (Of course, if the 401(k) has high fees or undesirable investment options, you will have to factor that in to your decision to roll over or not.)
