I think it's just unnatural. In speech you will hear people using it, when they think of a second subject as they talk:
I... and my father... went
This suggests to me that it's not a grammatical issue, but one of colligation, unlike the use oif 'me and John went', where there is a clearcut grammatical issue.