It's fine grammatically, but in terms of color true saffron is a more of a reddish orange when ground. The actual stigma from the crocus flower (which is where saffron comes from) is more of a crimson.
The yellow stuff is actually the cheaper alternative safflower.
You should know this - apparently Iran is responsible for over 90% of the world's true saffron supply!
Calling fabric 'saffron' might yield different mental images to different people, depending upon how much they actually know about saffron, but I don't suppose that really makes much difference to the story.