If my first ever Social Security check arrives in January 2024 because I am 70 in December 2023, is that income part of 2023 tax year (paid in arrears I believe) or 2024?

If your first SS check is sent in 2024 it will be income for 2024.   In January of 2025 Social Security will mail you a SSA1099 showing all of your SS benefits received in 2024, and you will use that when you prepare your 2024 tax return.

My understanding is that a SSA deposited in January is included in the new year's 1099-SSA, so 2024.