I've been thinking about phrases such as red shirt and shorts.
It can be interpreted two ways, one being both the shirt and shorts are red, or only the shirt is red right?
In many cases like this, common sense is the 'rule'. We tend to assume that if the two garments were not the same colour, then we'd say
either: He wore shorts and a red shirt. (We don't know the colour of the shorts.)
or: He wore grey (for example) shorts and a red shirt.
If we produce your original sentence, then the natural assumption is that both garments are the same colour, as bhai suggested. Note that he used the words 'the most likely interpretation'.