It depends.
First, you are correct on your interpretation of the rules regarding moving expenses and home office.
Second, it depends on if the "office" is considered your principal place of business or not. Commuting to your principal place of business is not deductible so if your office is the "principal" place then you cannot deduct the travel.
If your home office is your "principal" place of business then you could deduct the expenses to the office as it is a temporary work location.