If you never rented it out and it was always personal use, you would claim it as a second home. Losses are not deductible but gains are taxable. Yo enter it under Investment Income, stocks, mutual funds, etc., Say No to receiving a 1099-b or brokerage. Then choose second home, and enter your information. Each of you and your brother would enter half the amounts in your returns.
To get there, enter the words Investment income in search then click the jump To