Box 3 of the 1099-R form must remain blank. Nothing in an IRA is subject to capital-gains treatment.
If TurboTax is presenting a penalty on Form 5329, it appears that you have not properly indicated that you had $500 (not $463.78) of your contribution returned. From your description it's not clear if the contribution was returned from a Roth IRA or instead from a traditional IRA, so check the Form 5329 itself to see what penalty TurboTax is producing. Check the TurboTax worksheets related to IRA contributions and limits.
If you enter the contributions as if the $500 was never contributed, this will cause TurboTax to fail to prompt for the required explanation statement describing the return of contribution.
