Does it matter who makes income if filing jointly?

When I was entering W2's I accidentally had told Turbotax that is was my wife's W2. 

 

Later I saw that and changed it to me, but when I changed that W2 to be mine my refund went down by about $700.  I would think if I am married filing jointly it wouldn't matter whose name it is assigned to to calculate taxes.  

 

She also had a W2 but she works part time and makes a fraction of what is on my W2. Out of curiosity I actually changed both her W2 and my W2 to my name and again I got the higher refund amount. 

 

So it's only when I assign the W2's to the right person that my refund goes down.  Does that make sense?