Annualizing my taxes- will I get a penalty for not taking out taxes on my rmd, social security, and pension>?

I didn't take out taxes on my  RMD, social security, pension in the past years when I filed jointly with my husband and we owed about $200 - $350. Why is it so much when I have to file as a single person?