Can I deduct my home alarm system if I run a small business out of my home?

Yes, you can. Only the portion that would be allocated to the Home Office area of your business can be deducted.  If you purchased the system, enter it as an Asset (again, only the amount based on the home office percentage).  If you lease it and/or pay monthly monitoring fees, you will enter it in the same area as your Home Office.

does it get entered under utilities in the home office section?

Yes.  It can be entered there, or as an other Home Office expense.  Either way, it does get entered in the Home Office area as it is subject to the same percentage of usage as other common home office expenses.