Both are acceptable and meaningful, but they meaning different things. In 1. studies refers to school work, whereas in 2. study refers to a room in a house.
1. Tom should spend more time on his studies.
=> He should study more.
2. Tom should spend more time on his study.
=> He should clean up or finish building or designing his home office. Note, a study is a room in a house. Some people use it as a library of sorts, others use it as a home office.
All the best.