Some verbs can be followed by:
a bare infinitive: I can swim
a to- infinitive: I want to swim
a gerund: I enjoy swimming
a gerund or a to- infinitive with little difference in meaning: I like swimming/to swim
a gerund or a to- infinitive with a real difference in meaning: I tried swimming/to swim.
Unfortunately, there are no firm rules, though the core modals can be followed only by a bare infinitive. Native speakers rarely make mistakes.