You should try to get your employer to stop withholding CA tax, so you don't have to file for a CA refund every year, even if they are unable, or unwilling, to withhold NC tax.
Q. I have worked in SC, and AZ for them. I travel to states for different client sites. Do I have to pay taxes to those states.
A. Yes, but "nobody" does. See http://money.cnn.com/2013/06/28/pf/taxes/business-traveler-tax-threat/ If you live in a state without an income tax (e.g. FL or TX), it’s more likely you should file in the work states. You can't use the "it all comes out even" rationale for not filing.
