Your cost basis in retirement income (pension, IRA, 401K, etc,) is the sum of the nondeductible contributions to your "retirement plan/account" minus any withdrawals or distributions of nondeductible contributions. There are two types of contributions that one can make to a retirement plan, pretax contributions and after-tax contributions.
A pretax contribution is one made with money that is either deducted on your tax return (such as a Traditional IRA contribution) or is deducted from your gross pay through an employer (such as a 401(k) contribution).
An after-tax contribution is one that is made with money that has already flowed through your tax return and does not result in a tax deduction, such as a contribution to a Roth IRA or a nondeductible contribution to an IRA or 401K.
An after-tax contribution creates "Basis" in your retirement account. Because that money has already been taxed, when you withdraw it the amount is not taxed again.
Example: You contributed $4000 to the plan over the years(your cost basis). You took a distribution of $5000. Only $1000 should be state taxable, the first $4000 is a return of your costs. The program will ask for your cost basis of this distribution and will do the math.
For more information, see the IRS publications below.
