Yes. Although "I'm off" usually indicates you're leaving for a significant amount of time, so unless you spend an hour drinking milk, it would probably sound funny if you said "I'm off to drink milk." It's also quite common just to announce, "I'm off," and then if someone protests, "Oh, must you leave so soon?" you can then explain your intentions: "Yes, I have to meet someone for dinner." or "I have an early meeting tomorrow, so I have to get some sleep."
Student or Learner