Only the first $10,000 from your IRA is exempt from the early distribution penalty. If your spouse has an IRA they can withdraw up to $10,000 without penalty for the first time home purchase.
