Employer for 5 years has been withholding on a married status incorrectly for 5 years. I have always been single when I file. Do I need to refile all my taxes for5 years?

Employer Error. 

I have been with the same company for 5 years. Upon new benefits provider start this year I noticed that my paycheck withholding is listed as "married". The issue is I have not been married for 5 years. It was never updated.

Do I need to notify the IRS? I have always filed my taxes as "single" as I am "single". Would all my previous tax years need a review? How would this affect my tax returns? And yearly refunds?

Thanks!