I am trying to come up with a reasonable explanation for the grammar.
for me to go out = tenseless clause functioning as an adjective complement; it defines in what respect is the crowd too large --> the complement that completes the meaning of 'too large', an adjective (hence the name 'adjective complement'), is adverbial in function.
(too) large (for me) to go out
(too large) place (for me) to go out
Adjective complements do not complement noun phrases. In 'adjective complement', 'adjective' refers to the fact that an adjective is being complemented.