A couple of asides:
In my memory of the song (as sung by Petula Clarke back when I had a crush on her
, so I'm talking about something I haven't heard for a good few years) it was 'All the noise and the hurry' (repeating the rhythm of 'life is making you lonely').
"with 'em" is an internal rhyme with "rhythm". As the scene evoked is some kind of night-club, the 'them' probably refers to other patrons dancing. At a push, one could say that the 'them' might refer to the instruments or the musicians who were playing the
bossa nova - which would make the 'them' refer to something that had at least been mentioned in the previous line - but I don't think it's worth worrying too deeply about the sense; the lyricist was just looking for the rhyme, and probably didn't know what he or she meant anyway.
b