"to begin with" is usually used when listing a number of things
1) to make a point or 2) to make a retort to an argument.
1) To begin with, he has no job. Then he was late for our date. Finally, he has no money.
2) To begin with, you should never have taken the car.