I lost $5000 in earnest money when I decided not tp buy a property and another $6000 in upgrades can I take that as a loss anywhere

No. You cannot deduct lost earnest money deposits. IRS states this on their website:

"Nondeductible payments.   You cannot deduct any of the following items.

• Insurance (other than mortgage insurance premiums), including fire and comprehensive coverage, and title insurance.

• Wages you pay for domestic help.

• Depreciation.

• The cost of utilities, such as gas, electricity, or water.

• Most settlement costs. See   Settlement or closing costs  under Cost as Basis, later, for more information.

• Forfeited deposits, down payments, or earnest money."

What is the house I was buying was for investment, means I was buying this house for my rental portfolio, I own several rental properties, this was one more. I back out of the deal and lost upgrades money and earnest money. (the house was close to a landfill) .. in this case can I declare a investment loss? and how I enter on the system?