An RRSP is a tax-deferred account, so cost basis is irrelevant (in both Canada and the U.S.). 100% of the withdrawal is treated as ordinary income, basically like withdrawals from a traditional IRA in the U.S.
If you are a resident of the U.S., your RRSP manager will automatically deduct 25% non-resident withholding tax to remit to the Canadian government, before transferring the remainder to the bank account you provide for that. Just like in the U.S., where your total income is added up before taxes are calculated, you have to report the full amount of the withdrawal, converted to U.S. Dollars, to the IRS on your tax return. (I use the exchange rate that the bank applied to the net amount on the date I received it.) Then use TurboTax to calculate the foreign tax credit you can apply against the U.S. tax due on the full amount of the RRSP withdrawal. The credit will be prorated based on the percentage of your total reported income that was from non-U.S. sources. Chances are you won't get a credit for all of the Canadian tax, since U.S. tax brackets cover a lot of income before they reach 25% (and that's a marginal tax rate, not flat like the Canadian withholding). In principle you can carry leftover credit forward to apply to future foreign income in the same category on Form 1116, but in practice you probably won't have a chance to use it due to the permanent tax rate disparity.
Note that Canadian federal tax payments can only be credited toward U.S. federal tax. You'll have no credit against state tax unless you had provincial tax withheld from the withdrawal as well. As a non-resident, this would be very unusual. Not all states permit this, so if that is the case for you, you'll need to check on it. California in particular is notorious for being different (e.g. they don't allow deductions for HSA contributions).
