Did you report each sale on your tax returns? Since 1997 there is a new rule, you can't rollover the profit each year. So your 1997 and 2004 sales should have followed the new rules.
For a primary home, if you owned and lived in your house for 2 out of the last 5 years when you sell you can exclude the gain up to $250,000 for single or 500,000 for married from tax. You can not take a loss on your tax return. If you made more than a 250,000 (500,000 for joint) gain then the amount over it is taxed. Doesn't matter what you did with the proceeds like buy another house or pay off the mortgage
If you had a rollover profit from the sales before 1997 you had to include it on the next sale.
IRS pub 523 house sale. Figuring Gain or Loss on page 8.
http://www.irs.gov/pub/irs-pdf/p523.pdf
