Capture lab · Sunnyvale Baylands · 10 Oct 2026
Three unsynchronised drones, solved into one 3D ride
Lab 0 · Three views in space
Each video hangs where its drone was
The three real videos, placed at the camera poses the solver found and played on the recovered clock, with the 3D skeleton between them. The camera flies into each drone's seat: when it arrives, that drone's image and the skeleton line up. Freeze it and drag to look around the instant.
Lab 1 · Flythrough
The whole ride, solved in metres
The rider's skeleton, all three drones and the ground they mapped, on one clock. Ride along as any drone, chase the rider, or watch from above. Gaps are stretches the solver rejected. The ground is USGS aerial imagery placed by GPS; the solved path lands on the dirt trail it was ridden on.
Formation around the rider
Rider at centre, heading up. Each dot is a drone at its bearing and distance. The shaded wedge is the widest angle between drones: the bigger it is, the better the 3D.
Now
Lab 2 · Replay
The cleanest stretch, seen from all three drones
Switch drones off to see what fewer views give. With two drones the skeleton is still recovered but less precisely; with one, every depth along the drone's rays fits its own image equally well, and only the other two views can tell which is right. Rings in each video mark where the solve says the other drones are.
From the 3D skeleton alone: rider speed, torso lean from vertical, knee angles and their cycle rate.
This stretch
4D replays
Four windows you can walk around
The athlete rebuilt from all three drones and painted with their own pixels, inside a field baked from the same footage. Orbit, follow, or sit in any drone's seat.
Pipeline research · 11 Oct
Faster, no hand fixes, and the angles no drone saw
Each 20 s window now goes from three raw drone videos to a finished replay with no settings typed in by hand. The work below made that faster and more reliable, and tested a generative model for the views the drones missed.
One 20 s window, end to end (RTX 5090)
| Take | Before | Now | How |
|---|---|---|---|
| T1 · 4K30 | 442 s | ~225 s | 2D pose in fp16 on the GPU, body fit at 25 fps, bike off the critical path |
| T5 · 4K100 | 506 s | 266–280 s | processed at 50 fps with frame-exact decoding; camera-solve thresholds scaled to match |
| Disk per run | 7.5 GB | 2.4 GB | 4K frames deleted after export |
Reliability
Camera solve. A single mapping of T5 lost one drone on about 3 runs in 5. Two mappings now run in parallel and the pipeline keeps the one the body agrees with (fewest frames a held-out drone rejects). T5: 6 clean runs out of 6.
Clocks. If a drone's best offset lands at the edge of the search, the search widens (1.5 → 3 → 6 s). T1 with drone C's clock withheld: found −3049.22 ms; the hand-pinned value is −3049.18 ms. T2 ran with no clock hints at all (drone C +2.32 s).
Generated views where no drone looked (4DAnyone, benchmark only)
4DAnyone (Ant Research) turns one video of a person into synchronized videos from other angles. We fed it drone A only, rendered the camera drone B or C actually had, and scored it against that drone's real footage. Its fast weights are licensed non-commercially, so this is a benchmark, not part of the product.
| Variant (T4 255, held-out B and C) | Silhouette IoU | LPIPS | Time / 4.8 s window |
|---|---|---|---|
| As published (its own monocular body estimate) | 0.402 | 0.343 | 276 s (original code) |
| + the other drone's real view as reference | 0.406 | 0.342 | — |
| + our skeleton triangulated from the other two drones | 0.500 | 0.333 | 143 s |
| Same, 2 denoising steps instead of 4 | 0.495 | 0.332 | 101 s |
| Same, half resolution | 0.478 | 0.344 | 63 s |
What we learned. Measured geometry is what helps a generative model most: our triangulated skeleton raised silhouette overlap by 24%, more than any appearance change. Kernel tuning (FP8, compiled graphs) made it slower or ran out of memory on this GPU; the remaining speed has to come from fewer tokens and fewer steps.
Every angle
Same moments, three points of view
Four moments across the ride, each row one instant seen by A, B and C on the recovered clock. Drones occasionally catch each other in shot.
Lab 3 · Clocks
Find drone C's clock yourself
The drones log their own timestamps, and they disagree by seconds. Drag drone C's offset and watch the skeleton from A and B land on the rider, or miss him. The curve is the error at every offset; the solver picks its minimum.
Lab 3b · The motion is the clock
The same pedal strokes, seen by three drones
Each lane is one drone's view of the height difference between the rider's two ankles, which swings with every pedal stroke. As recorded, the drones' own timestamps put the strokes out of step. Shifted by the recovered offsets, they line up stroke for stroke. Right: drone B's offset measured independently in each segment, with the line the fusion uses.
Lab 5 · Coordinated formation
Where three coordinated drones would have flown
Today each drone followed on its own. Here the same path is replayed through our formation planner (receding-horizon dynamic programming over the formation's heading, after Ho et al., IROS 2021): three slots 120° apart around the subject, replanned every 0.5 s from the subject's current position and velocity only, avoiding the trees the drones mapped, within speed and acceleration limits. The dashed drones in the flythrough are its result.
Widest angle between drones
Measured only where the subject's 3D path is known. A simulation on the real path, not a flight: wind, GPS error and battery limits are not modelled.
Lab 4 · GPS vs vision
Where each drone thought it was
Wide bands: each drone's own GPS. Thin lines: its path solved from video. Arrows: from the solved position to where GPS put the drone, per segment, drawn ×2. Each drone's GPS is off by its own fixed amount, mostly in height; the shape of its path agrees with the solve.
GPS error by drone (median over kept segments, m)
Attitude log vs camera solve
Each drone's logged orientation, against the orientation solved from its pixels. Electronic stabilisation was on and the logs still hold to about a degree.
Quality
Every segment, graded
The take is solved in 35 s segments. Each is graded by hiding one drone and predicting where the rider's joints appear in its view (held-out error, worst drone). Segments that fail are rejected, not smoothed over. At the drones' typical 4–8 m range, 10 px is about 4–7 cm.
Every number on this page
What failed, and why. Five segments had a drone whose camera solve formed its own separate map, because it faced a different background from the others. Two lost the rider in one view. Lined-up drones were rare in the stretches that solved: 1% of frames had less than 15° between any two drones.
4K30 vs 4K100. The shutter was 1/2000 s either way, so frame rate did not change blur. Clocks and geometry decided quality today. A controlled test (a 100 fps take decimated to 33 fps through the same pipeline) is next.
Next flight. Hold about 120° between drones, hand off on turns, and start each take with a short hover facing a shared landmark. That fixes the separate maps and lets one battery cover more of the ride.
Aerial imagery: USGS National Agriculture Imagery Program (NAIP), public domain, via The National Map.