Should I be worried about the CP05 letter I got in the mail
There is no need to worry. Please see below for details on why you received a CP05 notice. And also see here Understanding Your CP05 Notice on the IRS's website. You don’t need to take any action at this time. The IRS will verify the information and either issue refund or contact you within 45 days.