I rolled over my 401k into a Roth IRA
the 401k issued a rollover check that was deposited to the new RothIRA. It was return on their system the next day.
I contacted the 401k ( issuer) and they maintain the check was cashed by the receiving company. The money has disappeared into thin air between these two companies.. it is not reflected in my new or old account. How do I inform the IRS I never received or was able to transfer any money, will I still be liable for a penalty on money I never received.
Informing the IRS is the least of your problems. If the amount in the account is substantial, I think you need a lawyer to help you recover the money. Once you get it straightened out, if the money ends up in the Roth IRA you might not have an IRS problem at all. Tell your lawyer to stay on top of how the 401(k) will report the rollover, and to make sure they don't report it twice.
If the amount is not large enough to hire a lawyer, you can still contact your state attorney general, department of commerce, or whatever other government agency that regulates banks and retirement plans in your state. It sounds like you need to gather proof from the 401k (like a copy of the canceled check, or transaction details of any wire transfer) and then go back to the new IRA bank. You might even contact whatever local news channel does "8 on your side" or whatever they call their consumer news and help segments.
Also note: you may be told that in order to do this kind of rollover, it must be completed within 60 days. And if you do get a check reissued, or the new bank finds the money, they may tell you that it is still do late to do a rollover. While the 60 day rule is generally true, there is a procedure to request a waiver for cause, such as a bank error. So if the money is found, you should still be able to treat it as a rollover.
It seems that there was a proper distribution from the 401(k), so I don't think that it's possible to avoid taxation of the distribution.
Depositing it into a Roth IRA as a rollover is a transaction that is not permitted to be undone. What was the reason for the Roth IRA custodian rejecting the deposit?
If you had used mobile deposit with the receiving custodian's app, the check would still be in your hands.