Modal verbs indicate something called "modality" -- they indicate ability or possibility. Examples are: can, will, shall, may, must, ought to. The first four also have past tense forms could, would, should and might. Note that "shall" is uncommon in British English and almost never used in American English, but "should" is common in both.
These verbs are sometimes called "defective verbs" because they are incomplete: none of them have infinitive forms, for example (you cannot say: "to may"), or -ing forms.
An auxiliary (or "helping") verb is a verb that helps out with the grammar. In particular, you use auxiliary verbs to make negatives. In this sentence:
I have finished my homework.
"have" is an auxiliary verb. It doesn't really mean anything, but we need it to construct the present perfect. To turn this sentence into a negative, we just put "not" after the auxiliary verb:
I haven't finished my homework.
The following sentence has no auxiliary verb:
She plays piano in the dark. (*)
To turn this into a negative, we need an auxiliary. Because there isn't already one there, we use the verb "to do" as a so-called "dummy verb":
She doesn't play piano in the dark.
Warning: Many auxiliary verbs lead a "double life" -- they also function as main verbs, and then they have real meaning. For example, "to have" can also mean "to possess", and then it's a main verb:
He has a new car.
He doesn't have a new car.
This also explains the phrase "How do you do?" -- the first "do" is an auxiliary, which we need to construct a question; the second "do" is the main verb.
Examples of auxiliary verbs are: to do, to be, to have -- plus all the modal verbs. (That's why sometimes you'll read about "modal auxiliaries" -- all modal verbs are auxiliary verbs, but not all auxiliary verbs are modal verbs.)
(*) Normally, we would say, "She plays the piano", but I am quoting from a song. It's OK to say, "She plays piano", but it's not so common.