Instead of dividends I have received "Return of Capital" which is to be subtracted from the cost basis. At this time the total of the "Return of Capital" is more than I paid for the stock.
Generally speaking, your basis cannot drop below zero. If you have received payments in excess of the amount paid for the stock, that's a disposition of the stock subject to taxation. Please explain your situation in more detail.
Return of capital payments are often seen in the case of utility stocks, real estate investment trusts, or corporations which are paying dividends in excess of their earnings and profits. If that's your situation, you are required to recognize any excess return of capital payment as a capital gain distribution.
I believe that you report the cost basis as zero, not negative. As somebody else responded, once your ROC distributions exceeded your cost, you should have been reporting those distributions as capital gains all along. I'm not sure if they are considered short or long-term capital gains though, so definitely check into that. It's the same net result conceptually, if you reported a cost basis of negative $10, that's just like saying you made an extra $10 in capital gains when you sold the stock.
Quick followup, IRS publication 550 says it all and fairly clearly:
https://www.irs.gov/publications/p550
Under the 'Nondividend Distributions' section:
Basis adjustment.
A nondividend distribution reduces the basis of your stock. It is not taxed until your basis in the stock is fully recovered. This nontaxable portion also is called a return of capital; it is a return of your investment in the stock of the company. If you buy stock in a corporation in different lots at different times, and you cannot definitely identify the shares subject to the nondividend distribution, reduce the basis of your earliest purchases first.
When the basis of your stock has been reduced to zero, report any additional nondividend distribution you receive as a capital gain. Whether you report it as a long-term or short-term capital gain depends on how long you have held the stock. See Holding Period in chapter 4.