I believe you will be fine. There is a remote chance you could get a letter from the IRS. If so, just respond with a letter and explain like you did in your question.
