If I work for a lone contractor, and I use his tools and he directs the work am I considered self employed? Everyone I meet in the industry says yes. Are they right?

Also, is it illegal to work as self employed when I should be considered an employee. Whose wrist will the government slap mine or my boss?