Is it okay to say " There is a restaurant very close to my house only 2 buildings down from house." Should I say down from from my house or building?
If it's a house, you can say 'my house'; if it's a building, you can say 'my building'.
But you don't need to say "from my house" or "from my building". You can just say, 'There's a restaurant very close to my house, only two (buildings)(doors) [down the street][away].
If the building in between your place and the restaurant has more than one door, "buildings" would be a better choice.