Another way of phrasing the sentence is
To me, that building resembles an egg (but not, to me, an egg resembles that building).
I think you should structure your sentence in the same way; start with the actual thing you are considering.
I usually associate that building with an egg.
Another example, what feeling do you associate with autumn? or What feeling do you associate autumn with?
I think this is more difficult to explain. Both look okay to me. I suppose the first looks better as it doesn't end with a preposition.
I am not a teacher.