The biggest problem with taking a large lump sum withdrawal from your IRA is that you may be put into a higher tax bracket plus more of your social security may be taxable.
Another factor to consider is your rate of return. If your investment growth in the IRA is higher than the mortgage interest rate, then you will have more money in the long run by keeping the mortgage. If you have enough income and deductions to take your mortgage interest as an itemized deduction, that reduces your net interest rate (4% interest only costs you 3.2%, after accounting for the tax deduction).
Another factor is investment diversification. Your IRA may be invested in many different investments that spread out the risk. You might be invested in mutual funds composed of many stocks, or invested in different types of stocks and investments with different goals (growth, income, stocks vs bonds, domestic vs international, etc.). Paying off your house puts a large part of your funds in a single investment that is not easily converted back to cash.
On the other hand, there is certainly peace of mind to be living in a paid off home.
