there is no absolutely right way to compute the average mortgage balance. that's for the $750 limit for the year.
one way would be to add up the month-end mortgage balance for home A to this add the month-end mortgage balance for the months that home B was not a rental (not necessarily the same as the months you lived in it) take the total and divide by 12. if this is under 750K your home mortgage deduction is not limited. it would be the interest on A plus the interest on B times the months it was not a rental divided by 12 (or use days it was not a rental to prorate the interest). The rest would go to the rental.
another way is to use the average at the beginning and end of month rather than month end.
