No, it is not. The Social Security withheld in box 4 is not related to the amount of the tax refund you should expect. That is the amount that is going toward the time when you are 62 or older so you can collect Social Security benefits. You do not get that back as a tax refund.
The federal tax withheld from your paycheck in box 2 is what you may receive as a refund--or at least some of it. Nor everyone even gets a tax refund. There are a lot of variables that go into the calculation of your refund including how much tax you had withheld from your pay, your filing status, whether you have any dependents, the credits and deductions you are eligible for, etc. etc. Enter all of your income and all of your other information and let the software do the calculations.
Are you asking about Federal taxes that were withheld from your Social Security monthly benefits, and are shown on Form SSA-1099? If so, you only get some/all of that back if it exceeds the total amount of taxes due on all of your sources of income.
If Social Security is your ONLY (Joint ) income, it is not taxable and you should stop the withholding.