Is the IRA/SEP/SIMPLE box marked on the Form 1099-R provided by the annuity company?
You didn't say whether or not the account at Vanguard is an IRA. If it was not (and also not some other type of employer qualified retirement plan), it's not a rollover.
Assuming this was a valid rollover, as long as you did not indicate that the distribution was all RMD, during the entry of the Form 1099-R from the annuity company TurboTax will ask what you did with the money. To indicate a rollover you must answer that you moved the money to another retirement account, then indicate the amount that you rolled over.
