Can I write off all or part of my mortgage if I’m clergy?

Yes. You may claim the Home Interest Deduction and there would be no restriction or limit since you do not get a housing allowance. 

IRS LINK