I disagree somewhat with Mike. Some people really enjoy being pushed to their limits. A friend of mine regularly takes part in races involving obstacle courses, wading through mud, freezing water, clambering over high walls - the races are incredibly tough, tiring and hard on the body. She readily admits that she is pushed to her limit each time but she enjoys every second of it and has a huge sense of achievement at the end.
I would say that the phrase certainly makes it clear that something will be difficult or challenging, but I don't think that makes it negative.