Both sentences are correct. Which one to use depends on the context and your intended meaning.
"We went down to California" is the simple past, a simple statement of a past action.
"We would go down to California" may be conditional and requires more context. "We would go down to California if it wasn't so far." "We would go down to California if we could afford to." It might also be a form of past continuous, equivalent to "used to". "When the weather at home grew too cold we would go down to California."
Student or Learner