When you are talking about finished action, you can say having said, having stolen, having done
He admitted having stolen the money.
They now regret having gotten married.
But it is not necessary to use that form. You can also use the simple –ing for finished action.
He admitted stealing the money.
They now regret getting married.
When one action happens before another action, you can use having (done) for the first action.
Having found a hotel, they looked for somewhere to have dinner. Having finished our work, we went home.
We could also say After –ing.
After finishing our work, we went home.
But having said that," he added "I think he will do well.
He said: "Newcastle are not unbeatable, but having said that they are a formidable team.
Yeah erm Er but having said that, was Oh absolutely yeah yeah.
And having said that, they recognize the difficulties of the moment. Having said that I'll try and speak to this. Having said that, I can say no more. Having said that, however, we need critically to explore the pluralist perspective.