Both my wife and I are F1 non-resident aliens (both of us live in US for around 3 years, is my tax withholding allowance the same as when I am single?

I am asking because it writes "If married, but spouse is a nonresidential alien, check the single box" on W-4. It means I still need to pay with single tax rate right?