You should be fine. Many people use both a PO Box for certain purposes. That said, if you are questioned by the IRS, you will need to produce evidence that you maintained the home. Such things as utility bills in your name can help to support this.
