Yes, gains in the traditional IRA are taxable when converted to Roth.
The IRS does not seem to be too aggressive about charging the late filing fee for late filed Forms 8606, particularly when the tax year in question has not yet closed as would be the case for a 2020 tax return. I would let the IRS bill for any late fees rather than pay them up front.
