Yes, you're right.
A gold watch is made of gold.
A mud wall is made of mud.
Something that is "golden" is gold in color. Something that is "muddy" has mud on it. You will have to use some real-world knowledge to figure out whether it's the color, the appearance, the covering, etc.
And sometimes you have to have real-world knowledge about the first, too. If I say I wore a "gold dress" to a formal party, you can be pretty sure my dress wasn't made of threads spun from gold. Although, if I were a celebrity or some rich princess, you might not be sure either way!