"Choosing a larger education credit" keeps lowering my AOTC credit and refund amount

You have a 1099-Q entered. After deducting all the allowed room and board expenses, you still paid more than the tuition and fees so the program is trying to figure out how much of that to make taxable to give you the biggest credit.

 

You comment that it is a nontaxable distribution, if you mean all of it is nontaxable, the 1099-Q should not be entered.  IRS Publication 970, Tax Benefits for Education states on page 45:  Don't report tax-free distributions (including qualifying rollovers) on your tax return.

 

If the Q is truly non-taxable, remove it and that solves your issues. 

There are three things you can do with your Qualified educational expenses (QEE):

1. Allocate then to scholarships (so that the scholarship remains tax free)
2. Allocate them to the 529 distribution (1099-Q) so that it will not all be taxable
3. Use them to claim an education credit

TurboTax allocates QEE, in that order, but it doesn't do a very good job if you want something different. TurboTax allocates QEE, in that order, until you tell it otherwise.  It's best if you have some idea of what you want and the outcome expected, when you make your entries. 

_____________________________________________________________________________________________

Qualified Tuition Plans  (QTP 529 Plans) Distributions

General Discussion

It’s complicated.

For 529 plans, there is an “owner” (usually the parent), and a “beneficiary” (usually the student dependent). The "recipient" of the distribution can be either the owner or the beneficiary depending on who the money was sent to. When the money goes directly from the Qualified Tuition Plan (QTP) to the school, the student is the "recipient". The distribution will be reported on IRS form 1099-Q. 
The 1099-Q gets reported on the recipient's return.** The recipient's name & SS# will be on the 1099-Q.
Even though the 1099-Q is going on the student's return, the 1098-T should go on the parent's return, so you can claim the education credit. You can do this because he is your dependent.

You can and should claim the tuition credit before claiming the 529 plan earnings exclusion. The American Opportunity Credit (AOC or AOTC) is 100% of the first $2000 of tuition and 25% of the next $2000 ($2500 maximum credit). The educational expenses he claims for the 1099-Q should be reduced by the amount of educational expenses you claim for the credit.
But be aware, you can not double dip. You cannot count the same tuition money, for the tuition credit,  that gets him an exclusion from the taxability of the earnings (interest) on the 529 plan. Since the credit is more generous; use as much of the tuition as is needed for the credit and the rest for the interest exclusion. Another special rule allows you to claim the tuition credit regardless of whose money was used to pay the tuition.
In addition, there is another rule that says the 10% penalty is waived if he was unable to cover the 529 plan withdrawal with educational expenses either because he got scholarships or the expenses were used (by him or the parents) to claim the credits. He'll have to pay tax on the earnings, at his lower tax rate (subject to the “kiddie tax”), but not the penalty.

 

Total qualified expenses (including room & board) less amounts paid by scholarship less amounts used to claim the Tuition credit equals the amount you can use to claim the earnings exclusion on the 1099-Q. 
Example:
  $10,000 in educational expenses(including room & board which is only qualified for the 1099-Q)

   -$3000 paid by tax free scholarship***

   -$4000 used to claim the American Opportunity credit

 =$3000 Can be used against the 1099-Q (on the recipient’s return)

 

Box 1 of the 1099-Q is $5000

Box 2 is $2800

3000/5000=60% of the earnings are tax free; 40% are taxable

40% x 2800= $1120

There is  $1120 of taxable income (on the recipient’s return)

 

**Alternatively; you can just not report the 1099-Q, at all, if your student-beneficiary has sufficient educational expenses, including room & board (even if he lives at home) to cover the distribution. You would still have to do the math to see if there were enough expenses left over for you to claim the tuition credit. Again, you cannot double dip!  When the box 1 amount on form 1099-Q is fully covered by expenses, TurboTax will enter nothing about the 1099-Q on the actual tax forms. But, it will prepare a 1099-Q worksheet for your records, in case of an IRS inquiry.

On form 1099-Q, instructions to the recipient reads: "Nontaxable distributions from CESAs and QTPs are not required to be reported on your income tax return. You must determine the taxability of any distribution." 

***Another alternative is have the student report some of his scholarship as taxable income, to free up some expenses for the 1099-Q and/or tuition credit. Most people come out better having the scholarship taxable before the 529 earnings. A student, with no other income, can have up to $13,850 of taxable scholarship (in 2023) and still pay no income tax. 

_____________________________________________________________________________________________

Provide the following info (one student-dependent at a time) for more specific help:

• Are you the student or parent.
• Is the  student  the parent's dependent.
• Box 1 of the 1098-T
• box 5 of the 1098-T
• Any other scholarships not shown in box 5
• Does box 5 include any of the 529/ESA plan payments (it should not)
• Is any of the Scholarship restricted; i.e. it must be used for tuition
• Box 1 of the 1099-Q
• Box 2 of the 1099-Q
• Who’s name and SS# are on the 1099-Q, parent or student (who’s the “recipient”)?
• Room & board paid. If student lives off campus, what is school's R&B on campus charge. If he lives at home, the school’s R&B “allowance for cost of attendance” for student living with parents.
• Other qualified expenses not included in box 1 of the 1098-T, e.g. books & computers
• How much taxable income does the student have, from what sources
• Are you trying to claim the tuition credit (are you eligible)?
• Is the student an undergrad or grad student?
• Is the student a degree candidate attending school half time or more?