If you filed a joint return, you agreed to be equally and severally liable for the information on the return, including unpaid taxes or false information that results in penalties.  The IRS can collect from either one of you, or both of you, in any amount that is convenient for them. 

It would be surprising to me if, in this meeting, the IRS agrees to only go after your ex-spouse for the debt.  Normally they will not limit their collection efforts, unless he is negotiating a settlement.

Normally, any debts from the marriage would be part of your settlement negotiations and something your attorney will work out for you.  But even if the divorce agreement says your spouse agrees to pay the debt, this is not binding on the IRS.  They can still come after you if you have more money to attack, or are getting a bigger refund.  

If the IRS does hold you responsible for the debt you can file for innocent spouse relief or equitable relief. Or you can sue to collect from your ex, if that was your agreement.

https://www.irs.gov/taxtopics/tc205