It's a few too many. I assume it's where "he" spent his childhood, not where his father spent his childhood? The father is the most recently mentioned male person, so "he" in "where he spent his childhood" could refer to the father.
Try this: It's in the small house where he spent his childhood, the house built of stone by his father.