If we change the terminology, it's easier to see:
The cement that sits between the bricks in prison walls.
However, I would say that if you use "... of beautiful walls", then using the article sounds better. If you use "... in beautiful walls", it sounds better without.
The secret that hides in the cracks of beautiful walls.
The secret that hides in cracks in beautiful walls.
And no, I can't really explain why right now!