Can we claim our son as a dependent or relative on the 2016 taxes? in 2016, he was a college student; he turned 24; he lived in our house; he earned more than $4,050.

How about 1040EZ What line determines

1040EZ Line 4

Since he was age 24 in 2016 he can no longer be claimed as a dependent under the Qualifying Child rules where his income would not be a factor in claiming him.  To claim him as a dependent under the Qualifying Relative rules his gross income must be less than $4,050 in 2016.  Since his income was $4,050 or more then no one can claim him as a dependent.

So if he made $4,000 gambling but lost that much- does his gambling income then count towards the $4,050 max gross income

Yes

In calculating income for the $4050 limit, is this adjusted gross income found on line 37 of form 1040?

No.  It is total income as shown on Line 22 of Form 1040.