In most cases, you are correct that losses on a mixed-use vacation home are limited to the amount of rental income you earned, with the excess carried over to future years. However, the software might be allowing the loss because you meet the 14-day or 10% rule: if your personal use of the home was for less than 14 days (or less than 10% of the days it was rented), the IRS treats the property primarily as a business. In that case, you can claim a loss against other income if you meet the active participation rules.
If you did use the home personally for more than 14 days or 10%, double-check your entries in the "Rental Properties" section to ensure the personal use days were entered correctly.
