You question seems to imply that you were subject to an RMD from the qualified retirement plan for 2015. To confirm that, did you reach age 70½ in 2015? Were you no longer employed with the company, were more than 5% owner of the company, or the qualified retirement plan requires all employees to begin RMDs in the year the employee reaches age 70½?
Assuming that you were required to take a RMD from the qualified plan for 2015, the RMD was required to have been distributed before the rollover was performed, even if you would not have reached age 70½ until later in 2015, because the first amounts distributed from your qualified retirement plan in 2015 were considered to be RMD until the RMD is satisfied. Even though the rollover was a direct rollover, it was still a distribution from the qualified retirement plan.
Amounts that are RMD are not eligible for rollover. The amount of your qualified retirement plan RMD deposited in the IRA is now an excess regular contribution to the IRA to the extent that it exceeds your eligibility to make a regular contribution to the IRA. You'll need to work with the IRA custodian to make a return of contribution of any portion of the amount that was ineligible to be rolled over or treated as a new, regular contribution. If you will be treating any portion as a regular contribution, you'll need to get the IRA custodian to correct their 2015 Form 5498 to show in box 1 the portion that is regular contribution and to reduce the amount in box 2 by the amount moved to box 1, with the rest being distributed to you as a return of contribution reported on a code P 2016 Form 1099-R.
