Thanks for the follow-up. It is difficult to know what "Lee vs. IRS" refers to, although there is an old Supreme Court case (1981) of the United States vs. Lee, in which an Amish employer is held liable to collect Social Security taxes on his employees, even though the Old Amish are exempted from paying Social Security taxes themselves when self-employed, for religious reasons. See https://caselaw.findlaw.com/us-supreme-court/455/252.html
I am thinking that this is not the case the IRS agent was referring to.
But I confess to assuming that you were making money at this business. As you saw above, I said, "You can demonstrate that you have an ongoing business as a travel agent".
A key identifier of an "ongoing business" is that it make money (at least most years). Otherwise, the business starts to look like an effort to avoid income taxes on your other income while avoiding any self-employment taxes.
I am glad that you retained a local tax advisor. As I noted, we here at the Community are limited in how much we can know of a questioner's situation. Good luck.
