No, Virginia tax should not be withheld from your pay if you do not live or work in Virginia.
A few states tax remote workers, but Virginia is not one of them.
However, if you sometimes go to the company's office in Virginia, your pay for the days that you are in Virginia is Virginia income, and you do have to pay Virginia tax on that portion of your pay.