In general, yes. I must admit though that I have never heard of "ice lolly".
pudding / jelly
When we are talking about different kinds or when they are putting in some cups or bowls, we use plural forms. Otherwise it should be in singular forms.
We use plural form when we have more than one.
Is my concept correct?
It's a popsicle to you, Mike.
I am not a teacher.
Lolly is short for lollipop.