In general, you can deduct the mileage when you travel from 1 site to the other in the same day. Travel from home to either site at the start of the day, and travel from either site to home at the end of the day, is non-deductible commuting.
Separately, you are asking if you can deduct mileage to the church on days when you don't also work at your other job, if you establish a home office. You first have to determine where your main place of work is. This is covered in IRS publication 463 and 570. Even though pastors are self-employed for certain income tax purposes, they are common law employees for most other purposes including home office. So you have to meet 3 tests not 2: regular use, exclusive use, and the home office has to be for the convenience of the employer and not your convenience. (If there is office space at the church and you choose not to use it, you can't take a home office deduction.)
If your home office does qualify as your principle place of business for your pastor job, then you can deduct any mileage you drive for the same business or trade (i.e. the pastor job). You still can't deduct your mileage from home to your full-time job, other than as mentioned in the first paragraph.
And of course you need accurate contemporaneous records like a log book or logging app.
See chapter 4 of publication 463.
https://www.irs.gov/businesses/small-businesses-self-employed/home-office-deduction
https://www.irs.gov/uac/about-publication-463
