Is "rat hole" an idiom or should I take it literally.
I've heard it used idiomatically, to refer to a very detailed/specific discussion between a small number of people at a business meeting:
Let's not pursue that rat-hole and waste everyone else's time. Perhaps you three could stay on after the end of the meeting.
Informally, people in a meeting might just say 'Rat-hole!' to identify such a discussion when it starts.
(Of course, matilda's right though - in some contexts it might not be used figuratively at all.)
Another rat-related idiom that you meet (in BE at least) is 'rat-run' - a series of residential roads where traffic goes dangerously fast (often with sharp corners, junctions and narrow roads) during rush-hour to avoid hold-ups on the main roads:
Since the roadworks started, the road where I live has become a rat-run between the bypass and the motorway junction.