There certainly seem to be a limited number of nationalities that this can be used for. I would say that the most common are Englishman, Englishwoman, Scotsman / Scotswoman (though in the full sentence you would say "He is Scottish" not "He is Scots"), Welshman, Irishman.
Other than the 4 countries that make up the UK, I think the only country I've heard it attached to is France. "He's a Frenchman."
Yes, the first letter would ALWAYS be capitalised because it refers to nationality.
Interested in Language