I wouldn't say ' fine depending on the context' - which strikes me as a bit too accommodating. But it's certainly true that there are a few contexts that would make that usage acceptable.
Example: 'I apologized for being late... to Tom but not to Dick or Harry'
(But in that context you could equally say 'I apologized to Tom - but not to Dick or Harry - for being late '
Student or Learner