@ltap91 You identified the critical issue: what is your principal place of business (PoB)*. If your home is not your PoB, then your mileage is not deductible, except if there is more than 1 client in a day and the trips are consecutive. It's unlikely that a baby sitter would qualify for the home being the PoB.
In your Example: Client A needs a babysitter for one time on Tuesday 1pm to 5pm. Client B need a baby sitter for a one time 6 pm to 9 pm. Only the mileage between Client A and Client B is deductible. The mileage to the 1st client's home and the mileage from the last client's home back to your home are not deductible. Any trips in-between are deductible. An exception is allowed for out-of-town trips. There is no simple mileage definition of "out of town".
*To deduct mileage, even if self employed, the home needs to be the "Principal Place of Business". It does not need to qualify for the Home Office deduction (the 'Regularly and Exclusively' rule does not apply).
https://www.irs.gov/pub/irs-drop/rr-99-7.pdf
https://www.irs.gov/publications/p463/ch04.html
https://www.irs.gov/pub/irs-pdf/p587.pdf (page 3) POB defined
