We had hardly sat down to the supper when the phone rang. (We had only just sat down and were about to eat supper or had only just started eating when the phone rang. )
We can't stop for coffee now, we've hardly started. ( We can't stop for coffee because we've only just started.)
'Hardly' has the idea of 'almost not'.