So it sounds like they did do the NIA calculation and then translated the dollar amount into some number of shares (different than the number of shares originally purchased with the excess contribution if there were other amounts in the account over which the NIA was required to be calculated) having a current total value equal to the amount required to be distributed. That would be fine. As I said, ignoring transaction costs, if any, that would be equivalent to selling the same number of shares inside the IRA, distributing the exact adjusted amount in cash, and then repurchasing the shares outside the IRA.
However, stock is not usually bought and sold in fractional shares, so to do this with whole shares the return of contribution would have to be accompanied by some amount of regular distribution to account for the additional fractional share needed to make a whole share. If the in-kind distribution was able to consist of fractional shares such as is possible with a mutual fund, this wouldn't be an issue.
Given all of the complications, I would never suggest that anyone do an in-kind distribution for a return of contribution. It serves no purpose other than to avoid a round-trip sale and repurchase and to ensure the that transaction is completed timely such that the cost basis of the shares outside the IRA is the same as the distributed value.
