No. There is no need to worry as long as you filed and scheduled it before April 18th. Generally, the funds are withdrawn on the payment date you specified if your return has been accepted. Sometimes, it may take a couple more days on their end. If you still don't see the debit 7-10 days after your return has been accepted and your specified date, you will need to call the IRS e-file Payment Services at 1-888-353-4537, if this is a federal tax payment you owe. If you didn't specify a date, then April 18th is the default payment date.