I don't think there are rules as such. It is more a process that occurs (or sometimes fails to occur) spontaneously. When nouns begin to be used as modifiers, they normally require modification. You cannot yet say institution research. You must use institutional. But note that you can say research institution where once you would probably have needed to say institution for research.
As an adjectival usage becomes more common, there is a tendency for the modification to be dropped. I still strongly prefer instructional materials, but I believe the process has progressed to the point that some people would use and others accept instruction materials at least in AmE.
If the adjectival usage becomes sufficiently common the modification often disappears entirely, and the noun form acquires an adjectival meaning in addition to its meaning as a noun. Countless examples can be found in any dictionary.