The act of "seeing" happened before the act of "looking pale" in the past.
He talked to me as if he had been my father.
This is impossible, at least grammatically. The act of "being your father" happened before the act of "talking" to you.
Had he been your father, he would have continued to be your father in the present. "Had been your father" means he was your father at some point in the past, but stopped being so in the present. He either passed away or disavowed your existence as his son. In that case he could not have talked to you as your father since he did not maintain the quality of being your father at the time of talking to you.
He talked to me as if he were my father. (unreal situation in the past)
Is the correct way to say it.
The act of "looking pale" can exist long after the act of "seeing." The act of "talking to you as a father" cannot exist after the person had stopped being your father.
I hope I didn't mildly complicate it!