If you choose to receive your refund by direct deposit, your refund would not be delayed because of an error on your address.  If you choose to receive your refund by check, then your check might be delayed if the address is wrong.

The main reason for a correct address if if the IRS or State need to contact you.  They send a letter to communicate with you.  You might want to put in a Change of Address form (8822) for the IRS (if you need to also correct for State, you can check your State's website for their change of address form.

How do I change my address with the IRS?

https://ttlc.intuit.com/community/irs/help/how-do-i-change-my-address-with-the-irs/00/25940