One afternoon, three drones, 86 slices of data.

On the afternoon of October 17 we flew three consumer drones around one person at Baylands Park for about eighteen minutes, in two takes. The subject walked, jogged, climbed a rope-net pyramid three times, stepped onto benches and a picnic table, jumped, and went up and down grass hills and a levee. This page lists every slice of that footage and what it became, with the numbers, the protocol behind each one and what we have not solved.

The setup

Everything captured

Both takes are split into 86 windows of 4 to 42 seconds, each labelled by activity and ground type. Walking takes up most of the time, but there are three full rope climbs, six step-up windows, two jumps, and about seven minutes on slopes, hills and the levee.

Minutes and windows per activity, both takes:

What each part became

1. 3D body for both whole takes

The subject’s 3D body is measured on 90.6% of take A and 87.1% of take B, from two or three drones at a time. That is 24,315 measured body frames at 25 fps. Every frame carries a flag that says whether it was measured: short gaps are flagged so they are never mistaken for measured frames, and longer stretches where fewer than two drones see the subject are left empty.

The 30-second stretches that pass our quality check: 21 of 22 in take A and 13 of 14 in take B, with a median reprojection error of 3.2 and 3.4 pixels.

2. Cameras, clocks and drone paths

Without a calibration rig or shared clock, each drone’s time offset and camera pose come from the video. The recovered drone positions agree with the drones’ GPS to 0.5–1.1 m at the median. As an independent check we hold one drone out: the body solved from drones A and B alone, drawn on drone C’s own frames, lands 8.4 px (levee crossing) and 6.2 px (table step-up) from where drone C sees the joints, at the median.

3. The ground under the subject’s feet

We rebuilt the ground from the same drone footage and checked it against the USGS 3DEP 1 m lidar survey of the park (bare earth, public domain). The score is the ground height under each foot contact minus the lidar, after removing one constant offset per window.

90th-percentile error at foot contacts, median over four slope windows, with every method scored on the same contacts (those our measured ground covers). The motion-only baseline is TERRA, published research from EPFL (open source), run as a benchmark only.

On the slopes our ground is within 10 cm at the 90th percentile and 3 cm at the median. Measured ground patches agree with the lidar to 1.8–7.6 cm median absolute difference. One grass-hill window is a failure case for coverage: the drones saw only about half of the ground the subject walked on. The picnic-table windows wait on a tape measurement and are not used for any claim. Per-window numbers and the full protocol are in Ground inferred from body motion vs measured from video.

4. The park as a physics scene

The objects the subject used were measured from the footage and placed in a MuJoCo scene with the measured terrain: the rope-net pyramid (pole 3.87 m), the picnic table (top 0.80 m, benches 0.46 and 0.48 m) and a bench (seat 0.375 m). The rope net in physics is a simplified cable net placed on the holds the subject actually used. The scene loads and steps stably with a Unitree G1.

5. Five 4D replays

Five short windows were rebuilt as 4D replays you can view from any angle: the three rope climbs, the picnic-table step-up and the levee crossing. The fitted body mesh sits 15.4–24.6 mm from the triangulated joints, and the ankle 3.1–10.5 cm at the 90th percentile. Ground, net and table in the replays are measured from the same footage.

6. Robots: G1 and H1-2 motion, and a G1 policy on the real terrain

80 motion runs (16 minutes) passed our checks. The 77 runs that are not climbs were retargeted to the Unitree G1 and H1-2, and cut into 233 four-second clips for a G1 tracking policy, which then runs each clip on the measured terrain in physics simulation.

Share of runs that finish without a fall, current G1 policy on the real terrain:

MuJoCo physics simulation, Unitree G1 torque limits. 100 randomised runs per clip with pushes and sensor noise. No real-robot runs yet.

With equal training (same clips, references, training settings and 3,000 iterations; two training seeds each), a policy trained on the measured ground finishes 1.5 points more runs than one trained on flat ground (95% CI +0.5 to +2.4). The gap grows on hard ground: +4.4 on the steepest quarter of clips and +5.6 on the quarter with the most relief, slices fixed before the runs. Giving the policy a height scan of the terrain adds another 1.5 points (one seed).

The person’s own step-ups, scaled to a robot 1.32 m tall, are beyond the policy: 4.1% complete. For the picnic table we designed a step-up motion for the G1 instead: ground, the 0.50 m seat, then the 0.83 m top, with whole-body IK that keeps the robot clear of the measured table. Trained on the measured table, the G1 reaches the top in 77% of 150 randomised runs with pushes (93% without), with no support. The same training on ground guessed from the motion alone reaches it in 0 of 150: the guess has a ramp where the seat is, and the robot walks into the seat edge.

Hill-down is the next weakest. Several of the remaining falls are clips that start mid-stride at 4–6 m/s or descents faster than the robot can follow. The H1-2 motion still slips on slopes, and no H1-2 policy has been trained yet.

7. Rope climbs

Kinematically, the climb now keeps the G1’s hands and feet on real holds: 95th-percentile joint speed fell from 15–22 rad/s to 2.0–2.8 rad/s, and a limb is off the net in 7–9% of hold frames. That is a kinematic result, not physics.

In physics simulation, a G1 climbing policy completes the climb in 6 of 6 runs with a pelvis support force capped at 1× body weight (0.56× used on average) and 0 of 6 at 0.5×. Climbing without support is not solved. The net is a simplified stand-in: grasps are modelled as springs and the ropes do not collide with the robot.

8. Processing time

Processing time as a multiple of the footage length, on one shared workstation GPU (October 18). “Data” is synced cameras, the whole-take 3D body, ground alignment and quality checks.

All 86 windows

Time is from the start of each take. “Drones” lists the drones that see the subject for at least half the window. “Measured” is the share of 25 fps frames with a measured 3D body; rows under 50% are greyed.

Dataset totals

Policy runs: 14 evaluations (policy versions, on flat and real terrain) × 233 clips × 100 randomised runs. Climb runs: 3 policies × 4 support caps × 6 runs. Video and raw sizes are read from the original files.

Still open

• Step-ups. The table step-up works with a motion designed for the G1 (77% of randomised runs, one training seed, without the seat backrests). The person’s own step-ups, scaled to the robot, complete in 4.1% of runs.
• Climbing without support. The physics climb completes only with a pelvis support force, on a simplified net.
• Hands. The drones frame the whole body; fingers need small fixed cameras near the work.
• Breadth and ground truth. One park and one person so far, with no independent body reference such as motion capture or IMUs.
• Not measured yet. H1-2 policy success (none trained), tape-measured table heights, and end-to-end time when footage streams in during the flight.

Next: a capture day at a robot team’s own site; body and hands together, with the three drones plus two to four small fixed cameras around the work area; and more people and kinds of ground, including a session next to a motion-capture reference.

Robots are shown in MuJoCo physics simulation. Terrain reference: USGS 3DEP 1 m lidar, public domain. Motion-only terrain baseline: TERRA (EPFL), used for comparison only.