If you have not completed a new W-4 tax form in the past couple of years, now is good time to do so. Making changes to both of your forms to adjust the federal taxes withheld is a helpful strategy. When completing the W-4, complete Step 2 of the Form W-4 instructions to account for a working spouse. Underwithholding can occur if both spouses enter Married Filing Joint filing status in Step 1 and fail to complete Step 2. Enter '0' for tax credits and you can also enter an additional amount to be withheld per pay period in Step 4(c) for additional withholdings.
Using the estimate you have given of owing 4K that past two years, you can determine the additional amount needed by diving it by the number of pay periods you are paid. For example, if you are paid Bi-Weekly (26 pay periods) and you want to have an additional $4K withheld per year, you can enter $154 ($4,000 / 26 - Rounded up) on line 4(c) of the W-4 form. You can each make these changes to your W-4 so you are able to increase the federal withholdings based on your income.
Also, make sure that you are increasing your Health Savings Account (HSA) and Retirement Contributions to reduce you taxable income when possible.