This is always a messy situation. A rollover from a Roth IRA to a traditional IRA is not permitted, so this is technically a regular distribution from the Roth IRA and a new, independent contribution to the traditional IRA. The contribution to the traditional IRA.
You have until the due date of your 2018 tax return including extensions to obtain a return of contribution from the traditional IRA. You'll need to convince Vanguard that their handling of this as a direct rollover is erroneous because the traditional IRA was not eligible to receive this as a transfer and have them make a return of contribution from the traditional . You can then make a late rollover of the Roth IRA distribution under Revenue Procedure 2016-47 due to either financial institution error or due to the money being deposited into an account that you though was eligible to receive the rollover but actually it was not. https://www.irs.gov/pub/irs-drop/rp-16-47.pdf
