I'm with you on this one.
I tend to think about it differently though. The past is used to express unreality in the present e.g. your example 'If I were ...' So if I want to express unreality in the past, I have a problem - the verb needs to be a past form for two reasons 1) unreality 2) it happened in the past. If I use a simple past form, I can only express one of these. So I tell my students that they need to use a 'double' past tense or as it it better known the past perfect, which when restricted to time refers to the past of the past. This may sound confusing here but my students find this easy to understand.