The circumstances you describe (receiving SS benefits and only working part time) does not make you exempt from withholding. This is true whether your SS benefit are from retirement or disability.
What makes you exempt is if you had no tax liability last year, and expect to have no tax liability this year. That is, you expect to make less than $12,000 from your part time job.
