It probably varies from country to country, but in the US when the salesperson asks you such a question while you're still in the fitting room, they genuinely want to provide you with a garment that you like and that you will purchase. So, for example, if the trousers are too small, you are expected to be honest, and also specific. "I usually take a size 38 in the waist, but these slacks seem to be a bit snug...." The salesperson will then diplomatically tell you that sizes aren't necessarily universal, and some manufacturers make their clothes a little smaller, and will then bring you two pairs of trousers in graduating larger sizes.
However, if the garment is something that you simply don't like after trying it on, it is perfectly acceptable to tell the salesperson that you do not want it. The diplomatic method of dismissal is to place the blame on yourself rather than indirectly insulting their wares: "This shirt looked nice on the hangar, but it just doesn't suit me....I guess I'm going to have to look around some more...." (Even if you've already decided that you're not going to purchase anything from that store, it's always polite to imply that you're still going to look around a bit after that first fitting disaster. ;-) ) And by being honest with the salesperson in telling him/her that you do not like the garment, it gives them the opportunity to take it from you and return it to the rack right then, rather than having to retrieve it from the fitting room after you leave.
