"The removal was done with exact fractional shares. It was not a mutual fund, it was an ETF, and they removed fractional shares." That works then. (Your original post indicated that they were stock shares.) IRS guidance does provide that the return of contribution can be accomplished by a distribution of assets having a current value equal to the dollar amount required as a result of the NIA calculation.
The only case where original cost basis of an investment in a retirement account is taken into account upon distribution is when a distribution of employer stock shares from an employer-provided retirement plan like a 401(k) qualifies for distribution of Net Unrealized Appreciation as part of a lump-sum distribution under section 402(e)(4) of the tax code (which has nothing to do with IRAs). Otherwise, cost basis of a particular investment in a retirement account is meaningless.
