Even if you have a 4361, you are only exempt when performing a job that has clergy tasks. Generally, that means you perform services and sacraments and conduct worship services. I will put a quote from the IRS at the bottom and a link to the instructions. If you are not performing ministerial duties, you are not exempt from social security and Medicare tax even if you have an approved religious exemption. For example, a pastor hired to perform custodial services at a church is not exempt. A pastor would never be exempt at a secular job like working at Starbucks or Walmart.
A housing allowance must be designated in advance and in writing by your church, either through the denomination, or a local board of deacons, elders, administrative council etc. that represents the membership.
Who is your employer? It sounds like you are maybe a chaplain? I would have to do more research, and I really don't have the time to get into a new deep research project. You may want to consult with a tax professional who understands clergy issues.
This is also a very good organization with good information. Some denominations may pay for a basic membership, or you may have to join as an individual.
https://www.ecfa.org/ProductDownload.aspx?ProductID=361
Without more research, it would be my guess that if you are a college chaplain, and the college is your direct employer, the college can't give you a tax-free housing allowance. However, there might be exceptions. Also, it would appear that, even if you are assigned to your chaplaincy by your denomination as part of your ministry, any percentage of your job that is non-ministerial (such as teaching a class) is not exempt from social security and medicare tax even if part of your job duties are ministerial and exempt.
If you are being paid on a W-2 and the employer withheld social security and medicare tax in box 4 and 6 of your W-2, there is no way to get that back on your tax return. You would have to talk directly to your employer about your status, to see if they will or can honor the 4361.
Hope that helps, but feel free to post more details, since I may not really understand your situation or your question.
https://www.irs.gov/publications/p517#en_US_2023_publink100033572
Most services you perform as a minister, priest, rabbi, etc., are ministerial services. These services include:
Performing sacerdotal functions;
Conducting religious worship; and
Controlling, conducting, and maintaining religious organizations (including the religious boards, societies, and other integral agencies of such organizations) that are under the authority of a religious body that is a church or denomination.
You are considered to control, conduct, and maintain a religious organization if you direct, manage, or promote the organization's activities.
A religious organization is under the authority of a religious body that is a church or denomination if it’s organized for, and dedicated to, carrying out the principles of a faith according to the requirements governing the creation of institutions of the faith.
Services for nonreligious organizations.
Your services for a nonreligious organization are ministerial services if the services are assigned or designated by your church. Assigned or designated services qualify even if they don't involve performing sacerdotal functions or conducting religious worship.
If your services aren't assigned or designated by your church, they are ministerial services only if they involve performing sacerdotal functions or conducting religious worship.