The tax code provides the required calculation method and indicates that the earnings calculation must be done on the Roth IRA account as a whole. The contribution is treated as spread across all assets in the account; you are not permitted to cherry-pick assets within the account on which to perform the calculation. If your overall Roth IRA account balance gained about 8.7% between the time of the contribution and its return, your return of the $4,600 excess contribution will result in it being accompanied by about $400 of gains. The IRA custodian is generally responsible for doing the calculation:
https://www.law.cornell.edu/cfr/text/26/1.408-11
Note that you are permitted to obtain this return of contribution and avoid the penalty for 2017 only if by April 18, 2017 you either requested a filing extension or filed your tax return. If you did not do one of these, the deadline for obtaining the return of contribution is not extended to October 15, 2018.
The gains are taxable and (because, presumably, you are under age 59½) subject to early-distribution penalty on your 2017 tax return.
