The cost of a home security system would not qualify for a tax deduction unless you claimed a home office for a business like a child care facility.
In this case, if 20% of the square footage of the home was used regularly and exclusively for conducting a business, 20% of the home security fee could be deductible.
See also this IRS FAQ.