Yes, you can deduct continuing education costs, but you cannot deduct the cost of becoming a real estate agent (so your license and initial "school"). To be deductible, your expenses must be for education that either maintains or improves your job skills. Expenses to learn a new trade or job aren't deductible.
As to your fees associated with being a realtor... if your "business" is operational, you can deduct any business expenses that you paid, regardless if you have received any income for that year. Be careful, though... if you have loss for your self-employment for 3 consecutive years, the IRS may think it is your hobby and not business and may disallow your losses.