"It used to be so natural to talk about forever
But used-to-be's don't count anymore
They just lay on the floor, till we sweep them away..."
I would say it's more common in American English, but you might hear it in British English. As you pointed out, 'my former friends' is the more orthodox way of saying it.