No. Not filing in previous years should have no affect on the timing of when you get your refund. Very few filers are having their tax returns processed because the IRS was not processing tax returns for a while and just started back again earlier this week.
Are you tracking the status at the IRS tracking web site?
See this IRS Q&A for how the process works: https://www.irs.gov/refunds/tax-season-refund-frequently-asked-questions
