If my wife and I have multiple annuities and pensions, how do we fill out a Form W-4P so that the proper amount of federal taxes are taken out of each annuity/pension?

It doesn't matter which distributions generate the tax withholding as long as the total amount of tax withholding from all plans combined is sufficient.  Generally, estimate your overall tax liability by preparing a simulated tax return, then prepare Forms W4-P specifying percentages that will result in the generation that amount of tax withholding in total.

You can't just fill out one Form W-4P. You have to give a W-4P to each company or financial institution that sends you annuity or pension payments that you want to have tax withheld from. That's why it might be simpler if you can have enough tax withheld from one or two plans to cover all the tax you will owe, as dmertz suggested.