"My mom raised me and my brothers alone," I offered hesitantly.
"You go to college?" she demanded.
"Yeah," I replied. "Graduate school, too."
Do I understand right, that college is a place where you can get bachelor's degree and graduate school is where you get your Masters?
College is for undergraduate degrees, University is for graduate degrees. All of the universities that I know of offer undergraduate degrees as well as graduate degrees. If a school is referred to only as "College" then it only offers graduate degrees. You could refer to a university as graduate school as well, so you are correct.