Yes you are right.
The author is saying
1. that not many people travel between blue and yellow.
2. that if you go into yellow from blue, it is hard to get permission to return
3. that this (2) is made very obvious by the fact that trains going to yellow may have few passengers but the trains returning from yellow have even fewer passengers.