What concerns you about the IRS "coming after you?" Did you obtain the money illegally? Generally speaking, any time there is a banking transaction in excess of $10,000 the bank is required by federal law to notify the Treasury Department - not the IRS per-se. The IRS is but one division of many within the treasury department. The only reason it would raise flags would be if the Treasury Department has an reasonable cause to believe you obtained that money illegally, or used it for any illegal purposes.
The Treasury Department literally gets tens of thousands of these notifications a day. What would make yours stick out? Nothing I can think of.
Likewise, when you use that money to pay cash for a home, that too will trigger a Treasury Department notification. That notification will just be one of the several thousand they will receive on that particular day too.
