IRA distributions are never subject to SE tax. You paid the social security or SE tax when you made the contributions, they aren't subject to FICA or SE tax on withdrawal.
By the way, a private IRA can't be used as a non-taxable clergy housing allowance in retirement. For your retirement income to be a non-taxable housing allowance, it must come from a pension, 401(k), 403(b), or other qualified plan sponsored by your church or denomination. If you roll over the money into a personal IRA it is no longer eligible for the housing allowance treatment.