"You should speak to him perhaps" This is correct. instead of "Perhaps you should speak to him" This is correct without the comma.
Or the even more strange example, when Limiting adverbs have been used such as "only",like in the case "I eat meat only" instead of "I only eat meat". "I only eat meat is commonly used. "I eat only meat" is correct.