Do you and your brother both live in the house? If so, the two of you are just sharing expenses. There's no gift, and no gift tax.
You said you bought the house "with my brother." Did he pay half the cost (or half the down payment)? If so, you're not giving him anything by putting him on the deed. He already paid for his share of the house. You're just formalizing the fact that he's a half owner. So again, no gift and no gift tax. But if you paid the entire cost, and are the sole owner, then you would indeed be making a gift of half the house by adding him to the deed. You would have to file gift tax return, but you would not have to pay any gift tax unless your total lifetime gifts (to everyone, not just your brother) are more than about $5.4 million, including the value of the half of the house that you give him.
The other tax ramifications, which you haven't asked about, involve claiming itemized deductions for mortgage interest and real estate tax. You can only deduct what you actually pay, so if you and your brother each pay half, you can't deduct what he paid. Normally you have to be an owner of the house to claim the deductions, so your brother wouldn't be able to deduct what he pays. But if he paid half the cost of the house, and he lives in it, he would probably be considered an "equitable owner" for income tax purposes, even if he's not on the deed, so he would be able to deduct the share of the mortgage interest and real estate taxes that he pays.
