Self Employment taxes vs federal taxes

Have a new sole proprietorship - no employees - self-employed. 

 

This year I have been paying quarterly taxes... but I haven't been subtracting self-employment taxes. Do I need to do that with my quarterly, or just at the end of the year taxes?