Tax Penalty on withdrawing more the 10k from IRA to for down payment for first-time home buyer

My understanding is that my wife and I can each withdraw up to $10,000 from my IRA to pay for the down payment on a house without the 10% tax penalty from the IRS since we are first time homebuyers. If we decide to withdraw $50,000, then would we be charged the the 10% penalty on $30,000 (50k minus 10k minus another 10k) or would we be charged the penalty for all $50,000?