The process to claim exemption from withholding is described on page 2 of the W-4 form:
"Exemption from withholding. You may claim exemption from withholding for 2020 if you meet both of the following conditions: you had no federal income tax liability in 2019 and you expect to have no federal income tax liability in 2020. You had no federal income tax liability in 2019 if (1) your total tax on line 16 on your 2019 Form 1040 or 1040-SR is zero (or less than the sum of lines 18a, 18b, and 18c), or (2) you were not required to file a return because your income was below the filing threshold for your correct filing status. If you claim exemption, you will have no income tax withheld from your paycheck and may owe taxes and penalties when you file your 2020 tax return. To claim exemption from withholding, certify that you meet both of the conditions above by writing “Exempt” on Form W-4 in the space below Step 4(c). Then, complete Steps 1(a), 1(b), and 5. Do not complete any other steps. You will need to submit a new Form W-4 by February 16, 2021."
Notice that you have to claim that you will have no tax liability.
You complete a new W-4 with your employer to update your Federal withholding.
Under Step 3, as you increase the number of qualifying children ($2,000 each) or other dependents ($500 each), your withholding should be reduced.
Under Step 4b, if you expect to claim deductions in excess of the standard deduction, your withholding should be reduced.
If you have more than one job, note that Steps 3 and 4 should only be reported to one of the employers on a W-4, not all of the employers on the W-4.
If these issues do not apply to your situation, you may need to update your entries under Step 2, where you input information about Multiple Jobs or a Spouse's Job.
Be aware that following these steps will reduce your withholding. However, the steps may or may not reduce the withholding to zero. That depends upon your income and the values that you enter in the various steps.
A W-4 is used each time you want to change your Federal withholding on your pay.
These changes only affect Federal withholding, not your state's withholding, if you pay a state income tax.