So the second is correct. "I can't remember what my mother looks like?".
In an embedded question, there is no change in words order and verb tense.
And I don't understand why you use "what my mother ..."
In my onpinion, this sentence must be:
I can't remember how my mother looked like?