The credit is first applied to your tax liability first and 40% of the remainder is refunded.
If the credit was worth $1,000 but you owed tax of $125, the $1,000 would first go to your tax liability of $120, leaving $875. 40% of this is refundable (875 X .40= $350), leaving a refund of $350.
The According to the IRS:
“How much is the AOTC worth?...
It is a tax credit of up to $2,500 of the cost of tuition, certain
required fees and course materials needed for the attendance and paid during
the tax year. Also, 40 percent of the credit for which you qualify that is more
than the tax you owe (up to $1,000) can be refunded to you.”