I recently bought a house to flip at the beginning of October 2017 and plan to sale or rent out on 2018 how do I file expenses on taxes?

You only report it when you sell the home ... 

purchase price + cost of purchase + improvements + holding costs = basis     

sales price - (basis + closing costs) = profit or loss to be reported in the year of sale.

The only item you can deduct in 2017 would be the property taxes paid in 2017 on the Sch A and only if you itemize deductions.  If you don't then you can roll that into the holding costs for the basis total.

What if I don't plan to sale but rent out can I claim those deductions

Once you place the home into rental service the basis of the home is entered as an asset to be depreciated ... you will not deduct the expenses in one tax year.  Then annual costs like RE taxes, insurance and the like are expensed annually on the Sch E.  Read the IRS pub 527 to get educated on rentals.