This is a common problem when your MAGI falls in the phase-out range and your excess contribution gained value. It's easier to either go through the iterative cycle, do a binary search or use the Newton-Raphson method to determine the necessary amount (before making next request for return of contribution) than to try to derive the formula (which involves an infinite series) to do it in once pass.
An alternative is to leave the remaining excess in, pay the 6% on that, then after October 15 but before the end of the year make a regular distribution of just the amount of the excess with no adjustment for investment gains.
