"We withdrew $50k from an investment account as down-payment money for our new home."
In a nutshell:
If the investment account contributions were not taxable in the year of the contribution, then they are taxable in the year you withdraw them. It just doesn't matter what you used the funds for. Be it a family vacation, buying a home or paying medical expenses. The destructibility of what you used the money for stands on it's own merits.
If the withdrawal was from a non-tax deferred account where the earnings are not taxed until the year you make the withdrawal, then generally (with rare exception) a portion of the withdrawal is considered to be earnings and the earnings will be taxed in the year of the withdrawal. Of course, this assumes you actually have a gain on the investment and not a loss.
The sale of a primary residence has nothing to do with where any original down payment came from. If the house was your primary residence for at least 2 of the last 5 years you owned it, then you can exempt the first $250K of gain on the sale from taxation. If filing joint and you both meet the "2 of last 5" rule, then the first $500K of gain is tax exempt.
