1. I should never have let you leave this place.
2. I should never have allowed you to leave this place.
Both of these are fine. It may be that you'd be more likely to find "allowed" in contexts where the speaker was in a position to give permission to the other person to leave.
Thus a father might say to an errant son, "I should never have allowed you to go out this evening"; while a wife might say to an errant husband, "I should never have let you go out this evening".
(But of course, vice versa is also possible.)