If you set up direct debit prior to filing your taxes, don't worry. As long as you filed your return or extension on or before the April 18 deadline, and you selected April 18 (or earlier) as your withdrawal date, you won't get penalized even if your debit hasn't gone through yet.
Since you're filing close to the April tax deadline, it's very common for your tax payment to go through after the date you specified due to the high volume of tax returns pouring in.
We suggest keeping an eye on your bank account, and if you still don't see the debit 7-10 days after acceptance, you should call IRS e-file Payment Services at 1-888-353-4537 or contact your state tax agency, as appropriate.