The likely explanation is due to the way the tax table itself works. It is not a straight percentage, but rather a graduated rate where every $50 of income results in additional tax due. Its possible to see a small amount of interest can push you over into the next $50 bracket, which then causes an increase in tax liability.
For example, a single taxpayer with $40,049 in taxable income that enters a $2 interest income would actually see a $12 increase in tax liability. It might be easiest to see if you look at the tables themselves at the link below -