@bhJogdt
Roth IRAs are completely separate. If you want to make non-deductible contributions and you are eligible to contribute to a Roth IRA, you should do that instead of putting non-deductible money into a traditional IRA. (Also, any workplace plan like a 401k is completely separate, they are controlled by different laws and regulations.)
All your money in any traditional IRA at any bank or broker is counted for tax purposes as being in a single IRA (individual retirement arrangement). The broker doesn't know or care if you took a tax deduction or not. You keep track of your non-deductible contributions on your tax return using form 8606, which will be prepared automatically. You need to keep copies of any form 8606s from your tax return for the rest of your life, this is an exception to the general rule that tax paperwork should be kept for 3 or 7 years. Because you made a non-deductible contribution (money that was already taxed and you did not take a deduction), that portion of your IRA will be tax-exempt when you withdraw or convert the money. But the IRS will not keep track for you, you need to keep copies of any form 8606 that is filed with your tax return so you can prove the non-taxable portion of any future withdrawals or conversions.
Because all your IRA accounts are considered one account for tax purposes, and because you have a mixture of deductible and non-deductible funds, any conversion will be partly taxable (due to the pro-rata rule). Even if you open a new account with a different broker and only convert that account, the IRS adds all your IRAs together and calculates the taxable and non-taxable amount of the conversion based on the total of your deductible and non-deductible funds across all traditional IRA accounts.
Because you have deductible funds in a traditional IRA, the only way for you to do a true non-taxable "backdoor Roth" conversion, is to first convert all your traditional IRA funds to a Roth, and pay the income tax on the portion of funds that were tax-deductible. You can do that all at once or gradually over time.
Or, leave your traditional IRA with a mixture of deductible and non-deductible funds, and keep adding non-deductible funds. As long as you keep copies of all your form 8606s, a portion of your traditional IRA will be non-taxable when you start withdrawing funds after you retire.
