If you take the money out of an IRA, and this is your first house, then the 10% penalty can be excluded. First house means you have not owned a primary residence for 2 years before the purchase closing date.
If you took the withdrawal from anything but an IRA, you do owe the penalty.
