Not necessarily.  You can let the IRS know the change if you would like them to change where they send information to you.  The IRS usually only needs the information on your address so they can know where to contact you.  So make sure to put the address they can contact you on your tax return.  That will be what they need.