Is transferring some of your assets from an IRA with Fidelity to a precious metals IRA where you physically own gold or silver is still considered a tax deferred IRA ?

Check back. I will page @dmertz for this.

It depends on the type of assets that are being transferred. If the assets that are being transferred are "collectibles", then these assets are considered a distribution and not tax deferred.  At this point, these assets are non-deductible contributions to your Precious Metal IRA. Please select this IRS link for further information.

 

As far as coins and bullion, this IRS source defines what is a collectible and what is not.

 

The following coins and metals are not included in the definition of “collectible” under IRC Section 408(m):

• Certain gold, silver, or platinum coins described in 31 USC Section 5112. See IRC Section 408(m)(3)(A) for the full definition.
• Any coin issued under the laws of any state.
• Any gold, silver, platinum, or palladium bullion of a certain fineness if a bank or approved non-bank trustee keeps physical possession of it. See IRC Section 408(m)(3).

In retrospect, it depends on the nature of the coins and bullion.

As far as I know, Fidelity does not permit precious metals to be held within their IRAs, so to have an IRA that holds precious metal coins or bullion you would have to liquidate an asset within the Fidelity IRA, perform a trustee-to-trustee transfer of cash to a IRA custodian that will hold the coins or bullion in an IRA and purchase the coins or bullion within that IRA.  Be aware that it's the precious metals IRA that owns the coins or bullion, not you personally.  If the coins or bullion then ever come into your physical possession, it's a distribution.  

 

While you could do a cash distribution from the Fidelity IRA and roll the cash over to the new custodian, that would be subject to the one-rollover-per-12-months limitation.   A trustee-to-trustee transfer of an IRA (or a portion of an IRA) is neither a distribution nor a rollover so such a transfer is not subject to that limitation.