Deductions & credits
Let's go back to square 1 because I messed up.
First, you can't deduct the interest as mortgage interest, because it was not acquisition debt; used to buy, build or remodel your home. Your deductible mortgage interest is the percentage of debt that is acquisition debt. Suppose your original mortgage was $50,000 and was all used to buy your house. Your new cash-out mortgage is $100,000 and you used the other $50,000 for something other than remodeling. That means that 50% of your mortgage is acquisition debt, and 50% of the interest is deductible. Turbotax should help with this calculation.
Second, you can deduct investment interest. This is interest used to buy investments. This deduction is still allowed even after tax reform. However, all other investment expenses (advisory fees, record keeping fees, and so on) are no longer deductible. Also, interest is only deductible up to your investment income. Suppose you bought stocks that appreciated in value, but you didn't sell them. They also paid $100 in dividends. That $100 is your income, so you could deduct up to $100 of your interest. The rest of the interest can be carried forward to be used when you sell the stocks and have a reportable taxable gain. (You can also declare the gain early to deduct the interest but this is an advanced strategy with many implications and you need to see a professional to do this, let's keep this as simple as we can.)
In Turbotax, the deduction for investment interest is here (this is the Desktop program)
See also this article
Here's the indication that other investment expenses are not tax deductible.
Now thirdly,
Because you don't have a specific loan with only investment interest, you will have to deal with the tracing rules. The tracing rules require that you be able to prove that a dollar of interest was due to a specific purpose, in this case buying stocks. The interest on the refi mortgage has two purposes, the mortgage and the stocks. You will have to keep very good records to show exactly how many dollars of that interest are traceable to the stocks. If you start using the loan proceeds for other purposes (vacation, home improvements, other debt consolidation) then you get farther and farther away from being able to trace the dollars due to buying the stocks. The harder it is to trace a dollar of interest back to the stock purchase, the more likely it is that the IRS will disallow the entire deduction if you are audited.
