Yes, they are both wrong. "Ever" is usually used in the question form of a sentence. However, it is possible to use "hardly ever" or "only ever".
For example:
I don't really like traveling. I've only ever been to Germany, and that was when I was a child.
She's hardly ever been late with the rent. She's a model tenant.