Flags are literally used to represent something or to show support of something.
If you have a "freak flag" it means that you metaphorically have a flag that shows that you're a freak (you're strange, eccentric, weird, unconventional etc.). This can be literally manifested in some eccentric behaviour, mannerisms, etc.
If you fly a flag, you are displaying the flag. The character says that while the other person has a "freak flag", they do not display it so it's hard to tell that they're "a freak".
Please note that while "freak" can be used as a derogatory word, it can also be used positively to mean that you're unconventional.
"to fly a flag (for)" is also used as an idiom to say that you support something.
PS: your native language is listed as English...?