Some of RMD was after tax contribution. How to calculate taxable portion of RMD?

From what type of retirement account did you receive the RMD?

What is the code in box 7 of the Form 1099-R?

Is the IRA/SEP/SIMPLE box marked?

Typically, the distribution will be reported on a form 1099-R and box 2 will often show the taxable amount. In that case the difference between box 1 and box 2 is the untaxed amount. Also, often the codes in box 7 will indicate the taxability of the distribution, as mentioned by @dmertz .

In the case of a distribution from a traditional IRA when you have basis in nondeductible contributions, boxes 1 and 2a will both show the same amount and the taxable amount will be calculated on Form 8606.