That depends on whether the gardener is your employee or an independent contractor. When you hire someone to work around your house or yard, that person may be your employee. The facts and circumstances determine the proper tax treatment  This person is your employee if you set the schedule, control when and how the work is done, and provide the tools used. It doesn't matter if you hired the worker through an agency or if you paid him hourly, daily, weekly, or per job.  

On the other hand, if your gardener is an independent contractor, he'll set his own schedule, work independently, and use his own equipment. An independent contractor usually offers his or her services to the general public. 

If you determine the gardener is your employee, you'll need to file Schedule H, Household Employment Taxes, with your tax return (because you paid him at least $2,000 during the tax year). The checklist attached below lists the tasks you'll need to complete when you have a household employee. It's an excerpt from the IRS Pub 926, Household Employer's Tax Guide, which is your best resource for Schedule H information.

What do you do if they refuse to give you a tax id number?

What if the facts and circumstances are a mixture of what you listed?

For example, you set the schedule for the work, but the landscaper uses his own tools and decide how the work is done. Which items take precedence?

You should not hire them. It is illegal for them to fail to provide a tax ID number and it is illegal for you to hire them without verifying they are legally entitled to work in the United States.

This is from the IRS Publication on Sch H: 

If you hired someone to do household work and you could control what work he or she did and how he or she did it, you had a household employee. This is true even if you gave the employee freedom of action. What matters is that you had the right to control the details of how the work was done.

Workers who aren't your employees.

Workers you get from an agency aren't your employees if the agency is responsible for who does the work and how it is done. Self-employed workers are also not your employees. A worker is self-employed if only he or she can control how the work is done. A self-employed worker usually provides his or her own tools and offers services to the general public in an independent business.

You can find the full details at https://www.irs.gov/instructions/i1040sh#idm140200252329504  Search for

"Did you have a household employee?" and you'll see some examples. 

Oops, I jumped into the middle of this thread accidentally. I haven’t quite figured out this new forum. It appears there is no way to delete one's own post.

If the gardener works for other people also, I think the most likely scenario is that he’s self-employed and therefore not your employee.