I think this will depend on context. There are two in which I can see "my" being attached to a workman in this way - as you say, an aristocrat with a large staff of workers, or if you are having work done in the house:
My electrician has had to rewire the entire building.
If you want to come round and I'm not there, my carpenter will let you in.
It is a colloquial use that is not uncommon here. It merely means someone who you have employed.
- For Teachers