"how can IRS determine by looking at a 1099-R whether it is a RMD distribution or a normal distribution?"
If this is your only regular traditional IRA distribution in 2017, the amount that is RMD is automatically the amount the amount of RMD you were required to take. The first amounts distributed from your traditional IRA in an RMD year are deemed to be RMD until the RMD for that account is satisfied. The IRS is able to do the same calculation as your brokerage. The IRS knows the amount of RMD you were required to take by knowing your birthdate and seeing the FMV shown on the previous year's Form 5498 from the traditional IRA sent to the IRS by the brokerage.
