Generally, 'place' is much more generic, widely-used, more widely applicable.
Most sites are places, but not vice versa. You wouldn't substitute 'site' in the following:
"This is the place on the desk where I put my coffee cup."
"I can't find my watch, but it must be in some place or another."
A site, as in "building site" is generally a circumscribed area where some specific activity takes place or has happened:
a crash site, a crime site, a bulding site, a delivery site. Even though they are called 'sites', they are all places as well. The site where you deliver something is no different from the place where you deliver it - because of the wider meaning of place. Yes, you can call the area of the street a place, the suburb is a place, etc. Anywhere is a place.