We really need more context to be sure. The 'bird' might be a flying animal or , not very respectfully, a female human. The nest might be a small thing made of twigs and leaves or some form of human dwelling.
an old bird's nest - This could be the home of an elderly bird, or the old home of a bird - among other things.
an old nest of bird - This is not natural English.
a nest of an old bird - this means what it says, though it could be a jocular reference to and old woman's home.
Student or Learner