he rolled them in at the cost basis of the day they were put back in the IRA.
I'm not quite sure what you mean by that. Cost basis of any particular shares is the value of the shares on the date that they were purchases outside of a retirement account or the date that they were distributed from a retirement account. The date that the shares were put back into the IRA has nothing to do with determining their cost basis. Because the same shares must be rolled back to the IRA, they are required to take away from the non-retirement brokerage account the cost basis of those particular shares which is equal to the value that the shares had on the date of the distribution from the IRA.
and say they will do 5498's in May or June, showing a rollover of the market value (which was less than the market value of the original withdrawal) when the shares were moved back into the IRA.
That's correct, the rollover contribution shown on the Form 5498 will reflect the value on the date that the shares were returned to the IRA. Because this will look like it disagrees with the reporting of a full rollover on your tax return, you'll need to include an explanation statement with your tax return explaining that what would normally suggest a discrepancy is the result of in indirect in-kind rollover.
I have been using average cost in taxable cost for 20+ years, to much to keep track of, and we rarely sell anything.
Yes, I thought about that possibility. Shares distributed in-kind from a retirement account will normally be treated by the brokerage as non-covered shares, so their cost basis would generally not be averaged together with any covered shares. Assuming that's the case here, if the brokerage does not revert the average cost basis of your non-covered shares to what it was before the shares in question were distributed to the brokerage account, you'll need to track the correct cost basis for the uncovered shares yourself; the average cost basis of any of the other non-covered shares of this stock or mutual fund tracked by the brokerage will be incorrect. The rollover of all of the shares in the December distribution must make everything go back to the way it was before the distribution both in the IRA and in the brokerage account. Even though the brokerage might have reduced your basis by the average cost as a result of the rollover, it really has to be treated as a rollover of the same lot that was distributed from the IRA.
it will be the same as if I had sold the shares, then repurchased them with cash inside the IRA?
No, it wouldn't, because of the difference between the value on the date of the distribution and the value on the date of the rollover contribution. Selling them outside the IRA would result in a recognizable capital gain or loss, so even if you sold the shares outside the IRA and repurchased them inside the IRA at the same price, the result would not be the same as an in-kind rollover which is not permitted to result in any recognizable capital gain or loss.
would have been clearer is I had sold them in the taxable account and transferred the cash back into the IRA and bought the shares then I guess.)
That is not permitted. If shares are distributed in-kind from an IRA, the rollover of that distribution must be the same shares that were distributed, not any kind of substitute.
