No, you don't need to do anything. It is common to get fraud calls. Something to remember is that the IRS will always contact you by mail first.
Phone scams are so common now days that it makes you wonder what the world is coming too. Try to remember that most if not all government DEPT’s will not leave a robotic voice message and their phone numbers are toll free. So if you get a threatening call from the I RS and the return call number isn’t toll free it’s more than likely a scam.
Even if it is a troll free number, the IRS does not call you. They mail letters.
Not unless you want to call the cops and try and get them interested in a sting. Since the call can be made with just a name and phone number, it's not evidence of any kind of compromise of other personal information like bank accounts or SSNs.
In the world today it is probably a good idea to periodically check your credit reports, and you can sign up for credit "locks" or "freezes" at the different credit bureaus, for different fees, and you can set up email and text alerts on your credit cards and bank accounts for unusual activity. But a phone call by itself is not evidence of information theft, unless the caller knows your SSN.
The concerning thing was they had my name and address; but then in today's world that isn't hard to come by. They did not say anything about SSN.