"By" refers to an extent of time: "By that time the building was nearly finished"/"You should be on question 5 by now". All the time up to the point of speaking, something has been happening.
"At" refers to a point in time: "At that time, and not before, you can go home"/"The train arrives at 15.09"/"We go home at 16.00."
b