One of the requirements to claim a dependent is he/she must be a US citizen or a resident alien. You needed to amend all the tax years you claimed your brother as dependent. The amended income tax return must be mailed to the IRS for processing.
See, how to amend a return you already filed, link.
