ROth IRA backdoor while having regular IRA

In 2022 I put $6000 into IRA with $0 in it and did Roth back door conversion.

Did not realize that in 2021 I rollovered $500K from 401 to another regular  IRA.

As I understand 2022 back door conversion can't be re-characterized back to IRA.

So will I pay taxes on whole $500K or there is a way to avoid it?

Thank you