Should I be seriously worried if I've never filed a 1099-int?

Seriously worried?  No -- but this is a situation not to be ignored.

You are required to report all income, no matter how small, and regardless of whether you got a form or not.  Usually the IRS matches up the items you report on your return with documents they receive from financial institutions.  If there is an omission they should let you know -- eventually..

Two bright spots:

• some interest you received may have been tax-exempt;
• the amount of interest you received may not have changed your tax liability.

When the IRS finds out, though, you might get hit with a small late-payment penalty for failing to claim interest income. If the IRS sends a notice, you typically have to pay a penalty of 0.5% of the tax owed. This charge is per month after the tax deadline -- April 17 this year -- and it includes the last half of April and the part of the month that the IRS sends you a letter.

But don't wait for the IRS to act if you forget to include interest as taxable income. Simply send in an amended tax return for the years you omitted the interest.

Please see the following FAQ for information on amending a return you already filed:

https://ttlc.intuit.com/replies/3288565

so I'm only able to amend back 3 years? what about discrepancies past that?

If the income you accidently didn't report is less than 25% of the gross income reported, after three years the IRS cannot assess more tax. If it was more than 25% then they have 6 years.

You can file an amended return going as far back as you want, but you can't get a refund after 3 years (or 2 from payment if that is later). TT may only support going back 3 years, but that is TT not the IRS.

If the 1099-INT amounts are small, I think most people would just ignore it and get it right in the future. Don't know what "small" is. $10 certainly. $1000 no.

According to TurboTax article, Top 5 Myths About Tax Audits:  
For “substantial errors,” the IRS maintains it can go back six years and recommends you keep most records at least that long. The experts agree, though: If an audit is going to happen, it will occur in the latter half of the three-year time frame. “Audits generally always happen two years after you file,” Zinman said.   <a rel="nofollow" target="_blank" href="https://turbotax.intuit.com/tax-tips/irs-tax-return/top-5-myths-about-tax-audits/L5W989jzq">https://turbotax.intuit.com/tax-tips/irs-tax-return/top-5-myths-about-tax-audits/L5W989jzq</a>

so up until last year interest gained was less than 100, last year was like 108. I'm in the military and gonna see a tax specialist to help me, but I can't help but be worried.

Seeing a tax pro is a good idea, but I would not be very worried about 3 yrs of accidently missing $100 of interest. As was pointed out that could well mean no actual difference in tax due or a very modest amount ($10-$25 a yr?).

Don't worry about things beyond three years as long as the understatement of income was less than 25% of your gross income for that year (they have 6 yrs for that) and you did not intentionally misreport (they can go back forever for fraud: not going to happen for $100 of missed 1099-INT).