Back

Where drone operators lose the people they search for

Unity · eye tracking · vision-language models CNRS@CREATE, Singapore 2024 study, analysis 2026

Thirty people each flew a simulated drone over a ruined city for five minutes, looking for 30 people lying in the rubble, while an eye tracker followed their gaze. I replayed all 120 logged flights in the original simulator, frame by frame, to rebuild what each operator had on screen and where they looked.

That shows where each person was lost: never on screen, on screen but barely looked at, or looked at and still not photographed. Then three AI detectors, two vision-language models and a classic person detector, were asked on every frame where the people are, to see what a copilot could have flagged.

The operator's screen with each of the four interfaces: first-person view only, with a top-down map, with a third-person view and the map, with a green trail of the path already flown
The four interfaces the operators flew with. Dashed: the drone's camera view, the part that is replayed and given to the detectors.
A participant's screenshot from 2024 beside the same instant rendered again by the replay: the two images match
The replay matches what the operator saw: a screenshot saved in 2024 (left), the same instant rendered again (right).
64%of the people not photographed had been on screen
13% vs 10%of the time a missed person was on screen, the gaze was on them, against chance
13% at mostof the missed ones would an off-the-shelf detector have flagged
Bar chart of the 30 targets of a flight, on average, for all operators and by gaming group: never on screen, on screen never looked at, looked at under one second, one second or more without a photo, photographed
What became of the 30 people of a flight, on average: never on screen (grey), on screen but never looked at (blue), looked at for under a second (yellow), a second or more but no photo (orange), photographed (green).

Four flights, replayed

One per interface. Left: the drone's path over the scene. Right: what the operator saw, with their gaze and each detector's alerts. Three times the real speed.

Never on screen On screen, never looked at Looked at, under 1 s 1 s or more, no photo Photographed
First-person view only. A frequent gamer: 17 of the 30 people photographed.
With a top-down map. 21 people seen but missed: many came on screen, few drew the gaze.
With a third-person view and the map. The detectors flag mostly the near, large targets.
With an augmented-reality trail (in green) of the path already flown.