Does it matter? You mean in some writings it's okay to use "having said that" and in others it's not okay?
Yes, exactly.
Could you please tell me when in writing it is okay and when it is not okay to use "having said that"?
The appropriacy of this phrase depends on what we call the 'genre' of the text. Some written texts are written to sound 'chatty', as if the writer wants you to imagine that he/she is talking casually to you. In texts like this, it's quite appropriate to use language that is characteristic of casual spoken English.
As I said, I don't know about the GRE exam, so I'm reluctant to give an answer one way or the other about that, but in the Cambridge international language exams, it would be okay, if a little unlikely, to use this phrase in an article.
I really don't want to comment further without seeing the specific use of the phrase in the text because appropriacy does also depend on use of the phrase.