Yes, you will add all capital improvement costs to the original cost of your home to determine the depreciable basis for your home office deduction. As you mentioned, you will need to remove the cost of the land (since land is not depreciable).
Please click on this
link IRS
- Home Office Deduction for more information.