If you want to remove any ambiguity, you could restate it as either:
"He went there with all of his cousins," or
"He went there with some (of his) cousins."
But there is nothing at all wrong with what you've written.
(not a teacher, just a language lover)