The phrase "of friends" modifies the phrase "A group". It functions as modification. We can delete it without changing the core meaning of the sentence. So, let's do that and see what the grammatical subject is:
 A group of friends => A group is coming to my house.
"A group", a singular noun phrase, agrees in number with the singular verb "is".
Note, with phrases like, a group of, a host of, a bunch of, and so on, it's becoming more common these days for speakers to parse the phrases according to proximity: the closest noun agrees in number with the verb, giving:
 A group of friends are coming to my house.
Hope that helps.