Capital Gains on Multi Family used as Primary Residence

I have owned a two-family property for 3 years. We have used one unit as our primary residence but have always rented the second unit (50% of property.) If we were to sell the building in order to buy a single family home as our new primary residence, my understanding is that I would owe capital gains taxes on the proceeds of the sale as they are not "like properties" and the current property has not been used 100% as our primary residence for two years to qualify for a primary residence exception. Is that correct?