Paying penalties on early distributions of already-taxed Roth IRA contributions

If my Roth IRA contributions have already been taxed, why am I paying a (tax) penalty each year on early distributions (before age 59 1/2)?  Shouldn't that technically be all my money since I've already given the IRA their portion?