While it sounds like your brother wasn't an especially nice guy - it's not "illegal" to claim you as his dependents and your permission isn't required if you met the qualifications to be his dependent. It sounds like you may have.
