Federal tax initially increased, then went back to what it was before I entered back door conversion.
Can you please clarify what state you live in?
a true "Backdoor" Roth contribution, done correctly, is tax-free so your tax should not change.
In MA, I believe you have to enter your backdoor IRA contribution in a separate place, so it is not taxed when converted.
( Not a MA taxpayer ).
During the MA state interview, you can enter the amount previously taxed on the Taxable IRA/Keogh Distributions screen.
Enter the nondeductible contribution amount under Other contributions previously taxed by Massachusetts.
While the summary may show your back door Roth IRA conversion as additional income, this does not make it taxable income. What is most important is whether your MA return shows it as taxable income.
See: