Yes - I think 'they' would be the much more common reference. I think the plural 's' of trousers makes us use a plural pronoun, even though the noun (a pair) is singular.
Often, with phrases like this, such as 'a group of people' or 'a set of saucepans', we use the plural form of the verb - 'are'- despite the noun phrase being singular. I used to think that this shows that the speaker conceives the thing to be plural in nature. Similarly, we say things like 'the police are coming' and 'Chelsea are beating Liverpool'.
However, in the case of 'a pair of trousers', I guess that most people would consider this as a single item, so therefore the choice of 'they' doesn't make sense. Maybe, as I said, the presence of 's' overrides the concept of singularity in the speaker's mind.
Any further thoughts on this?