Will you owe taxes at the tax time?  That depends on several factors, your total income on the Joint Tax return and what is being taken out as withholding from his paychecks etc.  Is he claiming these children on his W4 as well?  If both of you are claiming these exemptions then less tax is going to be withheld and you may end up with balance due at the tax time.

Congratulations! Estimating the amount of withholding can be tricky with multiple earners in a household - for most people the IRS.gov Estimated Tax Withholding calculator will generate an accurate W4 for you to submit to your employer but in some cases it will result in over or underwithholding. Turbotax has a great selection of tax tools for forecasting your tax needs at https://turbotax.intuit.com/tax-tools/ Taxcaster can estimate your total tax refund (or amount owed), and you can compare that with the estimated amounts that will be taken from you and your husband's paychecks. There is also a w4 withholding calculator on that site. 
If you have a complex tax situation, publication 505 from the IRS has an alternate method of setting your withholding between both of your w4s. https://www.irs.gov/forms-pubs/about-publication-505
Without knowing your income and deductions, there's no way to know whether the amount of income withheld by the federal government from your husband's paycheck is going to be sufficient to cover the total amount of tax due.  

Thank you for your answer! He is not claiming them on his W4 but I will be double-checking for sure! 

Thank you again! 

I will look into these options. Thank you so much!