It would be disrespectful to call an older man "Gramps," unless you knew him well and were sure he would take it as a term of endearment. "Pop" is out of favor as a term for a father, at least in my experience. "Dad" is more common for a father. In fact, variations of Pop or Pappy are terms often used for grandfathers now.
Families vary in their names used, of course.