Can one use:
1-I met him alone a few times. It was only then that I felt close to him.
2-I met him alone a few times. It was only on those occasions that I felt close to him.
Possibly, though it could also mean that it was after meeting him a number of times that you felt close.