You and your mother cannot both claim your brother as a dependent. If you both claim him, that will definitely attract the attention of the IRS. The best solution would be if you can get your mother to agree to not claim your brother.
The tax rules for a dependent are not based on legal custody or guardianship. As far as the tax rules are concerned, the person that your brother lived with for more than half the year is the one who can claim him. From what you wrote, it sounds like that's you. If so, your mother is not legally allowed to claim him.
You can claim your brother as a dependent if all of the following are true.
1. He is your son, daughter, stepchild, foster child, brother, sister, half brother, half sister, stepbrother, stepsister, or a descendant of any of them. You obviously meet this requirement.
2. He must be (a) under age 19 at the end of the year and younger than you (or your spouse if filing jointly), (b) under age 24 at the end of the year, a student, and younger than you (or your spouse if filing jointly), or (c) any age if permanently and totally disabled. You meet this requirement.
3. He lived with you for more than half of the year.
4. He did not provide more than half of his own support for the year.
5. He is not filing a joint return for the year. If he's not married, he would meet this requirement.
6. No one can claim you (or your spouse, if you are filing jointly) as a dependent.
7. Your brother is a U.S. citizen, a U.S. resident alien, a U.S. national, or a resident of Canada or Mexico.
