Thirty people each flew a simulated drone over a ruined city for five minutes, looking for 30 people lying in the rubble, while an eye tracker followed their gaze. I replayed all 120 logged flights in the original simulator, frame by frame, to rebuild what each operator had on screen and where they looked.
That shows where each person was lost: never on screen, on screen but barely looked at, or looked at and still not photographed. Then three AI detectors, two vision-language models and a classic person detector, were asked on every frame where the people are, to see what a copilot could have flagged.
The four interfaces the operators flew with. Dashed: the drone's camera view, the part that is replayed and given to the detectors.The replay matches what the operator saw: a screenshot saved in 2024 (left), the same instant rendered again (right).
64%of the people not photographed had been on screen
13% vs 10%of the time a missed person was on screen, the gaze was on them, against chance
13% at mostof the missed ones would an off-the-shelf detector have flagged
What became of the 30 people of a flight, on average: never on screen (grey), on screen but never looked at (blue), looked at for under a second (yellow), a second or more but no photo (orange), photographed (green).
Four flights, replayed
One per interface. Left: the drone's path over the scene. Right: what the operator saw, with their gaze and each detector's alerts. Three times the real speed.
Never on screenOn screen, never looked atLooked at, under 1 s1 s or more, no photoPhotographed
First-person view only. A frequent gamer: 17 of the 30 people photographed.With a top-down map. 21 people seen but missed: many came on screen, few drew the gaze.With a third-person view and the map. The detectors flag mostly the near, large targets.With an augmented-reality trail (in green) of the path already flown.