We can't see your tax return, but you must have entered something incorrectly or answered a question incorrectly somewhere along the line. The full amount that you converted should be on Form 1040 line 4b. Line 4b should not be zero.
In your original question you said "given the conversion to the Roth, we should not owe any taxes." That's not true. A conversion from a traditional IRA to a Roth IRA is fully taxable. (The only partial exception would be, as VolvoGirl said, if sometime in the past you made nondeductible contributions to the traditional IRA.)
You also said "Our 2023 tax return also shows this amount." But you have not told us where the amount that you converted appears on your tax return. It should be on line 4b, but you said that line 4b is zero. It's not clear what's on line 4a. If the amount is on line 4a, but line 4b is zero, then, again, you made a mistake in an entry somewhere.
Unless the amount that you converted is somewhere other than line 4a, the IRS is probably correct and you should follow the instructions in the IRS letter to pay the additional tax that you owe.
