Shadow means dark shape. We associate Shadow with dark side of a thing as:
Shadow economy: business activities that are difficult for the authorities to find out about.
cast a shadow over/on something (=the bad effects/influence that make something seem less enjoyable, attractive, or impressive)
Example: The events of September 11th cast a shadow over the celebrations.
The shadow cast by something = the bad effects or influence of something = the unfavorable effect of something(more or less the same as you have contended)