Maybe this is how I write to him
I was being neighbourly when I give you my number to have a friendly conversation together. I never expected that you would start sending me disgusting and offensive messages. I cannot tolerate any such abusive language; therefore, I have decided to not contact you anymore and to block your number.
As an academic, one is bound by a strict code of conduct. I believe your abusive behaviour towards me and my family may be in breach of that code.
Furthermore, if you knock on our door again or if you threaten or intimidate me or any of my family as you did the last time, I shall not hesitate to call the police immediately.
Finally, my 3-year old daughter runs and walks like any normal child would. She does not make excessive noise; certainly not outside the hours of 7 am to 10 pm.
I do not wish to have further contact with you, so in future, discuss your problems with the building management.