I would say "refused to change my bill" but not "give me change." That sounds to me (American) as though I gave him $5 for my $3.20 purchase, but he refused to give me the $1.80 that was due me.
Of course, in context, it would be clear enough. "I needed coins for the meter, but all I had was a $5 bill. But the cashier refused to give me change." -- No ambiguity there.