How much does your paycheck have to be for federal taxes to be taken out?

I have noticed that federal taxes haven't been coming out of my paychecks, and I am wondering if I'm not making enough. My employer told me they think that is the reason why taxes are not being taken out. Also, is this going to penalize me a tax time? Am I going to have to pay taxes when I file because federal taxes are not coming out yet?