When I filed my taxes in Mar 2021 for 2020 year, I did not claim a child, as I just had a child in Feb 2021. Am I eligible for advance payments for the Child Tax Credit?

Yes you are eligible for the advance CTC payment  however if you don't get it in advance you can get it on the 2021 return you file next year.   EVENTUALLY  the IRS will open that portion of the portal where you can add the child for the advance payments ... see the info here under MANAGE PAYMENTS  :    https://www.irs.gov/credits-deductions/advance-child-tax-credit-payments-in-2021

You would never be able to list a child born in 2021 on your 2020 return.

You will qualify for the enhanced child tax credit on your 2021 tax return.  The IRS has said they will open a web site where taxpayers can add newborns, but that hasn't happened yet.  Remember that the advance payment doesn't change the amount of the credit, just when it gets paid, so you will get the full amount of the credit eventually.