If you are married filing jointly, and you were both 65 or older at the end of 2016, you will have to pay some tax if your Adjusted Gross Income (AGI) is more than $23,200.
The $32,000 figure that you are referring to is the "base amount" that is used in calculating whether any of your Social Security benefits will be taxable. You cannot completely omit your Social Security benefits when you do this calculation. You have to compare the base amount to your total income including half of your Social Security benefits. You also have to include tax-exempt interest. But this calculation only tells you whether part of your Social Security benefits will be taxable. It does not tell you whether you have to pay tax on your other income. It's possible for none of your Social Security to be taxable, but to still have to pay tax on other income.
