Apply the cap after. You are not limited on the amount of mortgage interest that applies to the rental portion of your mortgage (room in your primary residence). Calculate the rental portion of your deduction on the full amount of mortgage interest paid.
For your home section you must use the required limitation after reducing the overall mortgage interest paid by the deduction used for the rental portion.
Manually enter the information in both the rental section and the itemized deduction section instead of letting TurboTax do the calculations. This becomes difficult with the various calculations needed to apply the correct deductions. You should also adjust the full amount of the mortgage in the itemized deduction section to remove the percentage that should apply to the room being rented.
To make sure that I understand correctly, the below is the correct way to calculate the deduction?
When renting out a part of your primary residence, there is one screen that will ask you the "WRONG" question for your specific and explicit scenario. The question reads: "Percentage of time I used this item for this business in 2020 (e.g.; 80%)"
What you need to enter in that block is the percentage of floor space, not the percentage of time. The program already knows the percentage of time, based on the "in service" date you entered on a previous screen.