As I said in my answer, if you made a "return of contribution," the contribution returned would be treated as if it had never been made. The response from Vanguard indicates that you received a regular distribution (code J, Tor Q on the Form 1099-R) which does not negate the original contribution.
The only possible way that you could make some amount of new Roth IRA contribution for 2017 would be if you could successfully argue to the satisfaction of the IRS that, despite the coding on the Form 1099-R, the original distribution represented some amount of returned contribution that was adjusted for gain or loss as required by section 408(d)(4) of the tax code and the distribution was incorrectly reported by Vanguard as a regular contribution, but your chance of success is slim. If the IRS does not accept your argument that the distribution was incorrectly reported by Vanguard, any new contribution that you make for 2017 will end up being an excess contribution subject to penalty. Given the low chance of success and red flag this would raise with the IRS, the likely answer is that there is NOTHING YOU CAN DO to be eligible to make a new Roth IRA contribution for 2017.
