Blow hot and cold suggests that your relationship goes through positive and negative periods. It's used when someone, for instance, likes and idea at first, then starts to be less enthusiastic, etc. How about something like saying that you two have never hit it off?