Does the pandemic relief count against your taxes and what if you didn't have any taxes taken out of?

What do you mean by "pandemic relief"? Are you referring to the extra $600 of unemployment benefits from the federal government? That is taxable income, just like regular state unemployment benefits. If you did not have any tax withheld from the extra payments, you will have to pay the tax when you file your tax return, and you might have to pay a penalty for not having paid enough tax during the year.

The COVID stimulus payments are not taxable income.  There was no opportunity to have taxes withheld from the stimulus payments, since they aren't taxable.