First you must understand that by signing a joint return for any year, you take full and equal responsibility for everything in that tax return, and the IRS can come after you for any deficiency even after your divorce. In fact, if you have more money than your ex and are the easier target, the IRS can come after you exclusively for all the money owed.
The "innocent spouse" relief is a request to be relieved of the burden of a tax debt that occurred during your marriage, and can only be requested after the divorce. You can request relief because this was your spouse's debt and you didn't know about it, or you can ask for relief on an equitable basis, by arguing that while you did benefit from the misstatement of fact (in the form of a larger refund or lower taxes) you already paid your share of the debt and the IRS should stop collecting from you.
https://www.irs.gov/businesses/small-businesses-self-employed/innocent-spouse-relief
You say there was no misstatement in 2016 or 2017--there must have been a misstatement at some point. Taking a loan from retirement is not a taxable event. However, it becomes taxable if the borrower defaults on repayment. For example, if your spouse retired or lost his job in 2018, any outstanding principal would have been converted to a taxable distribution at that time (i.e. 2018).
If you filed joint returns in 2016, 2017 and 2018, but filed single in 2019, then the default and the deemed distribution must have occurred in 2018 or earlier, making the distribution a taxable event on one of your joint tax returns. You may need to track this down with the IRS before you can file the correct paperwork to request relief. If you were divorced in 2019 and your ex defaulted in 2019, then it should not be a joint debt, the distribution would only have been reported on a 1099-R in his name and would/should be on his 2019 separate tax return.
Did you receive part of his retirement in the divorce as a QDRO (qualified domestic relations order)? I suppose it's possible that you also received half the debt and the loan default was yours, not your spouse's. I'm not familiar with how that would work. Have you checked your IRS transcripts to make sure there is no 1099-R in your name from any past year?
https://www.irs.gov/individuals/get-transcript
