IRA contribution not tax deductible because AGI is over the limit

I didn't know about this rule when I made a contribution to my IRA but I'm wondering if IRS will charge me 10% early withdrawal penalty if I withdraw the money back to my savings account?