I have not had any experience with being the general contractor or builder of a home. You sound like you are doing the right things to get going on your project. A 20% down payment is normal for a regular mortgage but may not be enough for the construction loan. You do have some value as collateral in the land you own but the banks want to see something they can resell if need be.
All the internet stuff I saw on construction loans was put out by loan companies-and you are seem not yet ready to have them ringing up your phone. Good luck with your goals . @jlohani do you have any info/sources for him?
Disclaimer: Not a tax professional. Information gathered from internet links. Anything dated in June 2019 was posted in prior years and is before the 2019 limits and changes.