Q. Since it wasn't our "primary residence," but a "2nd home" and it wasn't used as a rental for income, do we need to pay a tax on the capital gain on the sale.
A. Yes. The capital gain on the sale of a "second home" or other personal use property (other than your principal residence) is taxable. A capital loss is not deductible. A loss on investment property would be deductible.
A bigger question, for you is: is $50,000 your capital gain? You said "After calculating all costs for sale and upkeep and repairs over the years, there was probably a net gain of about $50,000".
The cost of "upkeep and repairs over the years" is not deductible from your gain on the sale (nor are they added to the cost basis). Only "improvements" can be added to the cost basis. "Costs for sale" are deductible, including any repairs explicitly made, in the year of sale, to get the property ready for sale.
