Unfortunately, yes, it is all taxable (but see below).
Income reported on 1099-R is largely or totally tax-deferred. Therefore, when you take a distribution from your retirement plan as reported on the 1099-R, it is normally taxable.
Note that qualified plans in which you have a basis do not tax the basis when returned, but this is usually only a small amount of the distribution and has nothing to do with what you used the distribution for.
You may be thinking of the exception to a penalty for an early distribution from certain retirement plans. If you took the distribution early (for most taxpayers, before you were 59 1/2), you would be subject to a 10% penalty on top of paying income tax on the distribution.
However, if your distribution was from an IRA or similar plan (but NOT from a 401(k) or other employer plan), then you could escape the penalty on the first $10,000 of the distribution ($20,000 if married filing joint) if you were a qualified first-time homebuyer. See IRS Publication 590-B to see if you qualify.
