Yes only one of you should report the 4 dependents on one W-4 or you you could each report two on the W-4.
Go to this IRS website and use their Tax Withholding Estimator for each of you - https://www.irs.gov/individuals/tax-withholding-estimator
