Generally, what comes after an "of" will be a noun phrase.
Usually, I'd say only "when I was your age." ("Age" is still a noun here. "Your" will come before a noun phrase as well.) To me "When I was of your age" doesn't sound natural.
He is of an age when he should know better.