Your question is unclear. What kind of retirement money are you talking about?
In general, all income is taxable in one way or another. If you withdraw money from a traditional pre-tax IRA at any time before age 59-1/2, there is a 10% penalty for early withdrawal unless you meet certain partial exceptions. If you withdraw money from a Roth IRA, there is no tax or penalty if you withdraw your original contributions, but there will be a 10% penalty if you withdraw the earnings (or certain conversion amounts) before age 59-1/2
If you separate from service with your employer in the year you turn 55, you can withdraw from an employer sponsored plan like a 401k or 403b without paying the 10% penalty, but if you still have money in a different 401k plan from a previous employer, you will pay the penalty if you are under age 59-1/2 (the age 55 rule only applies to the retirement plan provided by the employer you are working for when you quit or retire).
If you retire from a company with a defined pension, there is generally no penalty if you retire early and start taking the pension, although the earlier you take the pension, the less the payout will be.