By your own description ("It was used as a primary residence several years ago and is now empty/a vacation home that is hardly used"), you clearly have personal use property (and a loss cannot be claimed), not investment property.
Had you rented it out for the last "several years", you could claim a partial loss (prorated for for business/rental use time).
