Decreasing your withholding increases your tax liability by the same dollar amount.
Do you have other income besides the SS? If so, your Social Security may be taxable, so if you choose not to have any tax withheld, then yes, your refund or tax due will be affected. Social Security is taxable on federal returns and on some state returns.
TAX ON SOCIAL SECURITY
Up to 85% of your Social Security benefits can be taxable on your federal tax return. There is no age limit for having to pay taxes on Social Security benefits if you have other sources of income along with the SS benefits. When you have other income such as earnings from continuing to work, investment income, pensions, etc. up to 85% of your SS can be taxable.
What confuses people about this is that before you reach full retirement age, if you continue working while drawing SS, your benefits can be reduced if you earn over a certain limit. (For 2019 it was $17,640— for 2020 it was $18,240; for 2021 it was $18,960. For 2022 it was $19,560 — for 2023 $21,240) For 2024, $22,320.
After full retirement age, no matter how much you continue to earn, your benefits are not reduced by your earnings; your employer will still have to withhold for Social Security and Medicare. If you work as an independent contractor then you will pay self-employment tax for Social Security and Medicare.
To see how much of your Social Security was taxable, look at lines 6a and 6b of your 2022 Form 1040
https://ttlc.intuit.com/questions/1899144-is-my-social-security-income-taxable
https://www.irs.gov/help/ita/are-my-social-security-or-railroad-retirement-tier-i-benefits-taxable
You need to file a federal return if half your Social Security plus your other income is $25,000 when filing single or head of household, or $32,000 when filing married filing jointly, $0 if you are filing married filing separately.
Some additional information: There are 11 states that tax Social Security—Colorado, Connecticut, Kansas, Minnesota, Missouri, Montana, Nebraska, New Mexico, Rhode Island, Utah, and Vermont These states offer varying degrees of income exemptions, but two mirror the federal tax
Changing your withholding has no effect on the tax you ultimately owe, it only changes whether you owe or get a refund.
The tax you ultimately owe (your tax liability) is calculated on your tax return. If you have less withholding than you owe, you will have to make a payment when you file. If you owe more than $1000, the IRS can assess an underpayment penalty with interest, because you are supposed to pay as you go. If you have too much withheld, you get the excess back as a refund.