Hoey has pointed out that "while it is quite usual to speak of my doctor or my accountant, it is not usual to speak of my carpenter or my
bricklayer", but isn't this just because we see the first two more regulary, or we don't change them as often as we would a carpenter, for example?

I would guess the Lord at the manor would say "my carpenter/gardener/etc.".