Freelancer w/ Multiple Clients possibly having Former Employer as Client (QBI Deduction forfieted?)

So I have been doing freelance work for a handful of clients for a handful of years now.

I was also doing part-time work as an employee (getting a W2 from the employer) for another single organization.

I'm planning on leaving the organization where I was an employee, but they are interested in me doing some work (not the full role) on a freelance basis.

I've seen some discussion about former employees doing work on a contractor basis and not being able to participate in the QBI deduction (such as this thread https://ttlc.intuit.com/community/tax-credits-deductions/discussion/hired-as-a-independent-contractor-by-former-employer/00/2635720 )

I could understand if I'm not able to deduct 20% of the income of this particular revenue stream from a client that is a past employer.  

However, I want to make sure I'm not also forfeiting the possibility of deducting 20% of the income from my other, existing clients (that aren't past employers).   

Ultimately, I want to make sure I'm not penalizing myself for doing work with my past employer.  (Basically losing money in the short term, as a result)

Any thoughts on this?

Thank you so much for your time!