People wouldn't talk like that. Sentences 1 and 3 don't sound natural; 2 and 4 sound rigid.
Additionally, in these sentences, 'the man' is redundant because you're already using the pronoun 'he'.
The following is saying the same thing
- "He was quite/really easy to talk to." | "The man was quite/really easy to talk to."
- "He was quite/very difficult to meet."
or -- "It was difficult to have/arrange a meeting with him/the man." (If you are talking about meeting appointments)