The circles represent the 5 continents. They are coloured blue, yellow, black, green and red. When they are placed on the white background of the flag they represent all the colours appearing on the national flags of the world. This is the explanation given by Pierre de Coubertin, father of the modern olympics
If you want lots more information go here
http://multimedia.olympic.org/pdf/en_report_672.pdf