What @DianeW777 said.
If you follow the original suggestion of subtracting the Canadian tax paid to come up with the "taxable amount", you are leaving taxable income out and circumventing the proper calculation of proportional taxation. If you then also complete a Form 1116 to get a Foreign Tax Credit, you are effectively doing both a deduction and a credit. I suspect the IRS would frown on that.
Thanks to the tax treaty between the U.S. and Canada, a Canadian RRSP distribution is treated pretty much exactly like a distribution from a tax-deferred U.S. IRA would be - i.e. all of it counts toward taxable income (barring uncommon steps such as QCDs, which may or may not be permitted with foreign retirement income anyway). Moreover, when a foreign tax credit is claimed, that credit may only be applied against the tax that would have been due to the IRS on the amount of that particular foreign portion of your total income. Since Canada does a flat 25% withholding on RRSP distributions to non-residents, in most cases this will be higher than the applicable U.S. marginal tax rate, meaning that the credit from the IRS would be less than the amount withheld by Canada. So you can't simply take the total Canadian tax paid off the top when reporting it to the IRS. Note that the unused amount of the foreign tax paid will be available as a credit carryover, but you'll probably encounter the same situation next time, too.
