W4

For the past 2 years I’ve had to pay more than $2500 in federal taxes. I’m a retired teacher so I collect a pension. I also work a part-time job. What do I need to change on my W4 so I don’t have to pay again? Can I change a W4 on a pension?