You have to pay social security tax because the "retainer fee" you are paid is considered earned income, which if you're not an employee, is subject to self-employment tax.
