Three drones, one athlete.

On September 30 we took three small consumer drones to a park, flew them around one person for about forty minutes, and tried to turn what they filmed into a moment you can walk around. Here is everything: what we flew, what broke, the numbers, and the replay at the end. Nothing in our pipeline had seen a moving camera before this.

Why fly the cameras

The idea is older than this capture. In January 2026 we built SkyStudio at the Gemini 3 SuperHack in San Francisco: a simulation of 12 drones flying in formation around the field at Levi’s Stadium, ahead of Super Bowl LX, filming the players from every side and rebuilding them in 3D. It took 2nd place out of 45 projects. That was a simulation. September 30 was the first time we tried it with real drones and a real athlete.

Everything we had built before assumed tripods. Three cameras on stands, around a patch of ground, watching an athlete run through it. Fine for a drill. Sport does not stay in a box. A sprint leaves the frame in two seconds. A surfer is a hundred metres out. Tripods also put the cameras at head height, so the replay can only look from the side.

Drones fix all three. They follow the athlete, they hold any distance you like, and they can look down. The cost is that every frame now has a different camera position, nobody shares a clock, and the cameras are stabilised by software that bends the image. The test was whether the same reconstruction code runs on that at all.

The kit

Three DJI Neo 2 drones, flown with factory settings. We left every camera setting alone, on purpose. We wanted to know what stock footage gives us before asking for anything better.

The drones on the day. The low sun in the second photo comes back to bite us later. Drone 1 spent most of the session filming into it.

The day

We flew from about 4:30 to 5:10 in the afternoon at a park on the edge of the bay: flat grass, a tree line on one side, open marsh on the other. The athlete walked, jogged, sprinted, jumped and turned, and in one take threw the phone in the air to give us a rigid object to track. The three drones circled and followed.

Six stretches have all three drones recording at once. Together that is 926 seconds, a little over fifteen minutes of three-angle footage.

From the phone: the drones working into the sun, and a walk-through with them overhead.

Lining up three clocks

To triangulate a moving person, every camera has to be looking at the same instant. At jogging speed a tenth of a second is 30 centimetres. Our tripod pipeline syncs from sound. Drones record nothing worth using.

The telemetry looked like the answer, since every frame carries a GPS-disciplined timestamp. So we cut the three clips on those timestamps and ran the reconstruction. It failed. The athlete’s joints, rebuilt from two drones, missed the third by half a body.

The reason was that the timestamps are written when the log row is written, not when the image was exposed, and the delay differs per drone. We found it the way the tripod pipeline does its fine sync: by searching for the time shift at which the athlete’s motion, seen from two drones, triangulates most consistently. On take E the answer, relative to drone 1, was 677 ms late for drone 2 and 533 ms early for drone 3, with a clear single minimum (the best shift fit 2.5 times better than the next candidate). Solving take E a minute at a time, drone 2 came out 685 to 734 ms late and drone 3 486 to 531 ms early, so within a take the delay holds to about 50 ms (some of that is solver noise; the one-minute solves are less sharp than the 20-second one). Across takes it is a different story. Drone 3 sat 450 to 533 ms early in every confident solve, but drone 2 was about 700 ms late in takes E and F and about 950 ms late in B and D, which came from an earlier flight. Our read: the delay is fixed for a flight and changes when the drone is powered up again. Handy, because one confident solve per flight gives a tight starting point for every other clip from that flight, and the batch now uses exactly that.

Where the drones were

A moving camera has to be located on every frame. Features on the grass and trees are matched across the key frames of all three drones, and a global solver (GLOMAP) works out where every camera was and what the background looks like, all in one go, with no help from the telemetry. On a 20-second stretch with 201 key frames per drone (every third frame, used for the camera solve), all 603 of 603 landed in one map, and the background points, projected back into the frames, land within half a pixel of where they were seen.

It was not all clean. Drone 3 lost its place for the first 4.7 seconds (47 of its key frames never joined the map), and a sanity check against each drone’s own trajectory threw out 59 placements as implausible jumps and managed to re-locate 12 of them.

Then we used the telemetry as a referee. After fitting one fixed rotation between each drone’s gimbal and its camera, the logged attitude agreed with the attitude solved from video to 0.28°, 0.37° and 0.33° (median, per drone). Fitting each drone’s logged GPS track to its solved camera path leaves gaps of 0.36, 0.39 and 0.16 m (median). At the 3 to 5 metres the drones flew from the athlete, a third of a degree is about 2 cm. Two independent answers that close is the best check we have on the maps. The attitude check is the one that counts: GPS position residuals of 0.2 to 0.4 m show up on bad maps too, so they only catch gross failures.

The athlete in 3D

With the cameras placed and the clocks aligned, the athlete is the easy part: a detector marks 17 joints in every frame of every drone, and each joint is triangulated from the rays that pass through it. On the 20-second test the rebuilt joints land 3.8 to 5.7 pixels from the detections (median per drone at 1080p; a pixel is about 4 mm at that distance) and bone lengths vary by a median 6.7% over 589 of 599 frames.

Speed falls straight out of the 3D. In those 20 seconds the athlete covered 27.3 m and peaked at 8.9 km/h. The fastest moment in the whole capture is a sprint in take F at 17.1 km/h (take E peaks at 12.8). Both use the height scale from the aside above, so the sprint was more likely about 16 km/h (15.6 on the GPS scale).

From skeleton to body

You cannot render a stick figure, so we fit a surface. We fit a bendable 3D body template (Meta’s MHR, openly licensed, 18,439 surface points) to every frame at once: to the 2D joints as each drone saw them at its own exposure time, to the athlete’s silhouette in each drone, and loosely to the triangulated joints. One body shape is shared across the whole clip; only the pose changes. The fitted joints sit within 16 mm (median) of the triangulated ones, which is below the detector’s own error.

The template is a naked body and the athlete was wearing a loose polo. So the fit is refined: every point on the surface may move in or out until the mesh fills each drone’s silhouette and stops at its edges, with smoothness across the surface and over time. Silhouette overlap (1.0 would be a perfect match) went from 0.75–0.80 to 0.94–0.97 per drone. The model is also bald, which we learned the hard way, so the crown and back of the head get two centimetres of hair before refinement starts.

Drone 1Drone 2Drone 3

Colour from the footage

Our first attempt at colour averaged every drone’s view of each vertex into one fixed colour. It looked like a mannequin, right colours and no face. The fix was the renderer from our tripod lab, adapted to cameras that move.

For every pixel of the body, on every frame, it asks each drone where that point landed in its image at that instant, and blends their real pixels. A drone counts for more when its view is close to the virtual camera’s, when the point is well inside its silhouette, and when nothing blocks its line of sight (the renderer checks, for each drone, which parts of the body it can actually see). Nothing is generated. Sit at a drone’s exact position and the replay comes close to its own footage (21 dB on the test below).

Three things had to change for drones. The weights are measured relative to the closest drone, because ours were typically about 90° apart, at times nearly opposite each other, and a fixed blending rule left grey gaps between them. Each drone’s pixels are corrected for its exposure, measured on the grass all three could see (drone 1, filming into the sun, came out at 0.71 of the average brightness; drone 2 at about 1.35). And where no drone sees a point well, the renderer falls back to a texture baked from every third frame of all three drones, instead of to grey.

Ground, trees and shadow

The surroundings are rebuilt from the same footage: a ground map and a tree-line panorama, each baked from 270 drone frames with the athlete masked out. Because the drones keep moving, averaging many views also removes the athlete’s moving shadow from the grass. A second pass throws out any sample much darker than the average, which catches the rest.

The replay draws its own shadow rather than copying one from the footage. We know where and when we were (the drones’ GPS and clocks) so we know where the sun was: azimuth 249.6°, 20.5° above the horizon. The drones’ GPS also tells us which way is north in the map, and its idea of “up” agrees with the map’s to 1.4°. The shadow should land where the real one did. We have not measured that yet.

Hide a drone

We started by eyeballing screenshots and got fooled more than once. Now every change goes through the same test: hide one drone, render its view from the other two, and score the athlete against what it actually filmed. 79 frames across the three drones.

PSNR is a pixel-by-pixel match in decibels: higher is better, and every 3 dB halves the error. SSIM compares local structure on a 0 to 1 scale. At 15 dB the shape and broad colours are right but the face and shirt are smeared (look at Fig. 5 rather than the number). 21 dB is the same renderer allowed to see the answer. With the exposure correction on that cheating score drops to 19.1 dB, so the correction is doing something besides correcting.

The gap between the last two rows is the work. Where the error sits helps: 69% of it is inside the body and the render misses only 2.4% of the silhouette, so the shape is right and the pixels borrowed from the other drones are off by what looks like a few centimetres, enough to ghost the shirt and face. We think that points at camera positions and timing, not at the renderer.

1234

Does it hold on another take? Everything above was tuned on 20 seconds of take E, so we ran the identical chain, untouched, on 20 seconds of take F, from a different stretch of flying. It held up:

The image score is as good or better on take F; the body fit is looser, which shows in the outline. One thing we learned: for take F we first re-solved the cameras at ten key frames a second, as for take E, and the map came out worse than the batch’s three-a-second solve of the same minute (half the key frames rejected, joints off by 66 to 394 pixels). More key frames is not automatically better on fast drone motion. The take F result above uses the batch’s map.

The replay

Here it is. Follow the athlete, fly around, or sit in a drone’s seat to see the replay over the footage it came from.

The whole capture

The same pipeline is working through all fifteen minutes, one minute at a time, with thinner camera maps than the 20-second test (three key frames a second instead of ten). Pick a take below to scrub it with the three drones synced. 12 of the 15 minutes have a confident sync and show the field, drones and athlete in 3D. Takes A and C and two minutes of D are still fighting us; those play the footage only.

Speed

This is the uncomfortable part. On tripods we aim to process a clip in about six times its length. The first drone run took just under 15 minutes for 20 seconds to get cameras, sync and the 3D skeleton: 44 times real time. The replay steps add about ten more (body fit 7 minutes, refinement 2.5, colour and export a few more), so 20 seconds of replay cost about 25 minutes, roughly 75 times real time.

Over four fifths of it is working out where the cameras were from pictures of grass, which the drones’ own logs already tell us to within a third of a degree and a few tens of centimetres. Thinning the key frames (three a second instead of ten) brings a minute of footage down to 12–18 minutes for cameras and skeleton, but the maps get weaker: some drones then place only 45–100 of 181 key frames.

Then we fixed most of it. Overnight we went through it stage by stage, keeping the quality checks above as guardrails. The big wins were obvious in hindsight: decode only the frames we need, on the GPU (the old decoder walked through every frame before the start); match images only between drones whose telemetry says they face the same way (over half of the old pairs pointed more than 90° apart and never matched); and give the map solver each drone’s gravity direction from its attitude log, which also made the maps more reliable.

We re-ran the headline number ourselves rather than trusting the log: 123 seconds for the same 20-second clip, sync within 10 ms of the first run, joints and replay score within noise (table above). The replay steps came down too (body fit, refinement and export from about ten minutes to about four), so a minute of three-drone footage now costs roughly 18 minutes, down from 60–80. On 20-second clips the camera half hits our 6× tripod target; on 60-second chunks it is 8× (5.8× with two chunks in flight). The replay half is the next job.

What we learned

1. The pipeline survives moving cameras.Sync, mapping, triangulation, the body fit and the renderer all ran on consumer drone footage with their factory settings.
2. Drone logs place the cameras well but cannot time them.Attitude to 0.3°, position to 0.2–0.4 m, but timestamps off by half a second to a second, fixed within a flight but not between flights. Place the cameras with the logs, sync from the picture.
3. Screenshots lie.They flattered several versions. Our first test also let the hidden drone’s colours leak into the fallback; the fair version scored lower (14.7 dB against 15.6).
4. Remaining error looks like alignment.The silhouette is right to a few percent; the borrowed pixels are misaligned. The next lever is better camera positions and timing rather than a better renderer.
5. Speed decides whether this is a product.From 75× real time to about 16× on the same 20-second clip in one night (about 18× per minute of footage), against a target of 6×. A fleet that logs its own positions precisely would remove most of the camera solve, still the biggest single step.

The same motion on a robot

One thing we wanted to know: is motion captured this way clean enough to drive a humanoid? We took three stretches of the athlete from takes E and F, mapped them onto a Unitree G1, and trained a controller for each to follow them in physics (MuJoCo, real time, the robot free-standing with its real torque limits). Then we tried to knock it over: 59 extra runs each with random floor friction, body masses, motor gains, control delays and pushes every one to three seconds.

All three held up, with errors about half of what we saw on the public dataset’s sprints. That is mostly because this motion is easier: the athlete tops out around 3 m/s here against nearly 5 m/s there, and there are no pick-ups. The robot also runs a little flatter than he did (both feet off the ground 24–27% of the time against 40–44%), and it touches its torque limits at the peaks. The motion was scaled to his real height and the robot’s shorter legs, so it moves at about three quarters of his speed. All of it is simulation, compared against our own reconstruction rather than ground truth, and the controller is trained per clip. It does show that three consumer drones, flown without markers, captured motion clean enough for a humanoid to follow.

Simulating our own fleet

Before building anything we wanted to know what formation, how many drones and what hardware actually move the result. So we built a simulator and calibrated it on this capture. The measured inputs: joint detections are off by about 5.7 pixels at 1080p, and that barely improves when the athlete fills more of the frame; about 70% of that error is independent per joint; clock error left after our sync is under 5 ms. As a check, we dropped one drone at a time from take E. The simulator predicted the skeleton would move 23, 18 and 27 mm; the real footage moved 18, 18 and 31 mm. Close enough to trust it for comparisons, and everything below is simulated on real and synthetic motion.

A few findings, in order of how much they surprised us:

• Skeleton accuracy stops improving at three drones. With the drones kept 10 m apart for safety, more drones means a bigger ring and smaller athletes in frame. Extra drones pay off in body surface seen from two sides and in how much of a virtual orbit stays close to a real view: four drones see 47% of the body from two or more sides against 21% for three, five see 68%.
• Spread matters more than count. Two drones work best 90–120° apart (18.5 mm of change on Sep 30) and worst nearly opposite (31 mm). On Sep 30, drones 1 and 3 spent much of the time 145–170° apart.
• Where the ring points barely matters for the skeleton (1–4 mm). Turning it with the athlete is about keeping drones out of the running path and out of the sun. Turn it too eagerly and drones collide: following every cut within 0.1 s brought two drones to 0.6 m apart in the cutting drill; smoothing the heading over 2 s kept them 8.6 m apart.
• Feet were only cut off when a drone came within 3.3 m. Beyond 3.7 m it never happened on Sep 30.
• Formation choices hardly change battery life (under 3% at running pace). Flight time is set by the airframe.

Our plan from this: three drones about 120° apart at around 6 m, looking down 25–30°, switching to a wide arc above jogging pace; build four so losing one still leaves a full ring, and design the product around five. Ranking the hardware changes one at a time against the Neo 2 as flown, a 4K camera with tight crops on the athlete is the largest single gain in the simulation (57 → 33 mm), then formation control, then RTK and logged gimbal angles (57 → 49 mm, plus most of the camera solve gone). The 4K gain rests on the noise model, so the next check is running our detector on 4K crops of the Sep 30 originals.

What our own fleet would fix

Almost every limit above traces back to using consumer drones as they come. The table is what owning the fleet would change.

The cheapest next step is free: fly the same three drones again with stabilisation off, 60 fps, locked exposure, a clap at the start of each take and a deliberate 120° spread. That tells us how much each consumer default is costing before we spend anything on hardware.