If I fill out line 4a on my W-4, do I still need to fill out line 4c, or will they increase my withholding based on line 4a alone?

Yes, if you enter an amount on line 4a, Other Income (not from jobs) your employer should withhold more from your paycheck. 

They most probably will withhold for that additional income at the same tax rate they figured on the income you earned from them, depending on how much you enter for line 4a. 

HERE is an IRS tax withholding tool which you may find helpful.