"children's" modifies the noun "doctor". The "a" is not attached to "children", but to the noun phrase "children's doctor".
I was taught that in such phrases the a is attached to children.
You may not have been taught correctly on this issue. If you have a football game, where both are singular, what does the article connect to? Are you talking about a football or a game, with additional information about the type of game?
Infants are not children? Nyggus has given us a two word phrase with no context. We have no idea as to the the age of the children.