Depending on what it's the password to, all except 'unmindful' are possible.
"Remiss" isn't really a common word, outside of some phrases like "I would be remiss if I did not..."
"It [STRIKE]is[/STRIKE] was very careless of me to forget the password" and "[STRIKE]it[/STRIKE] It was [STRIKE]is[/STRIKE] very careless of me to have forgotten the password." Do both [STRIKE]the[/STRIKE] sentences have the same meaning?
'to have followed is, I suppose, technically wrong. As with my example above, few people will be bothered by it.
I think the following comment about the perfect infinitive used in a past tense sentence applies to the OP's sentence.
Not necessarily. You can say "It was very remiss/careless of me to forget my password".