The verb ought is a modal verb and this means that, grammatically, it does not behave like ordinary verbs. In particular, the negative is formed with the word not alone and not also with auxiliary verbs such as do or have. Thus the standard construction for the negative is he ought not to have gone. The alternative forms he didn't ought to have gone and he hadn't ought to have gone, formed as if ought were an ordinary verb rather than a modal verb, are found in dialect from the 19th century but are not acceptable in standard modern English.
Here, there's an example of the past tense of "ought". You see that "ought" itself has no past, you have to resort to the verb "to have" to convey the idea of past tense.
We ought not/oughtn't to have agreed without knowing what it would cost