I think you mean too, but look at the construction of your sentence and see if you could see how someone might read it as "I donít listen to people I donít respect or [don't] consider foolish,..." You can avoid that by inserting a "those."
I donít listen to people I donít respect or those I consider foolish,...
"Rather" young just means younger to some degree. If you're 60, you might consider someone who is 35 too young to have an informed opinion, but I wouldn't say that person was "rather young" unless you compared him to others in the retirement village.
Interested in Language