The pro-rata calculation is done separately each year. Whatever small amount of the first year's conversion is nontaxable reduces the amount of basis in nondeductible traditional IRA contributions that remain in your traditional IRAs by the same amount. The amount of basis that carries forward to the next year appears on line 14 of the Form 8606.
Until you have no money left in traditional IRAs, you'll always have some amount of basis in nondeductible traditional IRA contributions that remains in your traditional IRAs. Once you have distributed or converted all amounts from your traditional IRAs, all of your basis will have been applied as nontaxable amounts of distributions or conversions.
For example, if you have $5,500 in basis and you convert $50,000 (half of your traditional IRA balance) to Roth, leaving a year-end balance in traditional IRAs of $50,000, $2,750 of the conversion will be nontaxable, leaving $2,750 of basis on Year1 Form 8606 line 14. If you convert everything else the following year leaving a $0 year-end balance in traditional IRAs, the $2,750 from line 14 of the Year1 Form 8606 carries forward to line 2 of the Year2 Form 8606 to be used in calculating a nontaxable amount of $2,750 for that conversion leaving $0 on line 14 of the Year2 Form 8606.
