No, not at all. The only scenario I can think of for "left to work" is something like this:
I had loads to do at work today but people kept interrupting me. It was really annoying! Eventually, I put a sign on my desk that said "Leave me alone. I'm busy." Finally, everyone got the message and I was left to work.
Even in that scenario, I don't like the phrase. I'd say "I was left alone to get on with my work".