No, you do not have to have to declare that refund as "income" because you have already paid federal income tax on that amount.
That is, since you cannot deduct SS taxes or Medicare taxes from federal income and since the income they applied to was also (normally) in Wages in box 1 on the W-2, then you have paid income tax on them already and you never deducted them.
So the refund is "tax-free", because you already paid tax on it.
