This is from inversion english grammar
In formal English, it is quite common to use inversion after negative adverbial expressions and restrictive words such as only, never, hardly and little.
At no time did he get permission for what he was doing.
Not until the next morning did she realise how serious it was.
Only later did they learn his terrible secret.
Never before have I seen such awful behaviour.
Hardly had we walked in the door when the phone started ringing.
Little do you know how much trouble you are in.
Check out this site as well:
List of 18 Types of Subject/Verb Inversion