The interest paid on the $20K is not deductible. You have to had spent it on major improvements to your home. Using it to make mortgage payments doesn't count. You have what is called a mixed use mortgage consisting of both acquisition and non-acquisition interest (the $20K plus any costs of obtaining the refi that was rolled into the refi).
Is this loan a refinance of a previous loan: Yes
Did you use your loan for anything else: Yes
Have you used the money from this loan exclusively on this home? No
Since you first took out the loan, how much did you spend to buy, build, or substantially improve the home: Enter the amount of the original mortgage when you first bought the house.
For the purpose of calculating how much interest you can deduct, principle payments are applied to the non-acquisition interest first. However, since you have been paying down the mortgage for 4 years it is very likely that you have covered the cash out and are now left was only acquisition debt. If so the $20K cash out is no longer affecting the interest deduction and you are only with the overall mortgage interest limits.
