If you're single, you're not even required to file a Virginia tax return unless your VA adjusted gross income is equal to or greater than $11,950.  See this:  https://www.tax.virginia.gov/who-must-file

I'd like to get some of my taxes refunded though.  Additionally, according to https://www.individual.tax.virginia.gov/calculators/income-tax-calculator.cfm my taxable income definitely qualifies for tax ($120 + 5% of excess over $5,000)

If VA taxes were withheld from your pay, you're correct that you would have to file a VA tax return in order to get those refunded.

If you have to file a non-resident NY return, be sure to complete that first, before you do your home state VA return.  You can find NY's filing requirements for non-residents here:  

https://www.tax.ny.gov/pit/file/nonresidents.htm

I did the NY non-resident form first and already have some credit.  But regardless of that my base tax for virginia (before subtracting credits) is 0 despite my taxable income which is what I am confused about.