kldill1,
You have no recourse. The adult aged son is allowed to claim your son. He is related (brother, half brother or even step brother). Being related and living with the child is all he needs to qualify. He does not have to provide any support. The fact that you don't live with the child means that you can't claim the child without form 8332.
Essentially the child's custodial parent gets to decide who can claim her child. The non-custodial parent can only try to negotiate with her. I agree, it's nonsense. But, it's legal nonsense created by congress.
This may help in your negotiations: The special rule in the case of divorced & separated (including never married) parents applies to this situation. She can give you form 8332 to allow you to claim the the child's exemption and child tax credit and the adult son would still be able to claim him for EIC and HoH.
