If you actually did not perform the work in Virginia, you are correct that you don't need to file a Virginia tax return.
If your income was reported on a 1099-Misc, you would be considered self-employed and include the income on your resident state return.
However, if you were paid on a W-2 or did actually perform work in Virginia, you should file a Non-Resident Virginia return.
Prepare this return before your Resident Maryland return.
Here's more info:
https://ttlc.intuit.com/replies/4776808