-What had you been doing by 03:00 pm?
The past perfect continuous with 'by' is unlikely here. If we are interested in activity completed at or before that time, we might ask, "What had you done by 3 pm?"
If we are interested in activity begun before, and continuing up to, that time, we might ask, "What had you been doing until 3 pm?'
If we are interested in activity at that time, we might ask, "What were you doing at three pm?"