AutelicaRobotics data
Unsynchronised consumer cameras (2–5 exocentric views, 3 or more recommended, optionally one egocentric head camera) are synchronised and calibrated from the footage. Body, hands and head are triangulated in a metric world frame. Each clip includes a held-out-camera error report and a physics-tracking evaluation on a Unitree G1.
Our flight, Sep 30Motivation
Egocentric datasets for robot learning are labelled at scale by monocular 3D hand, body and head-pose estimators. A head-mounted camera observes the wearer's hands but little of the body, and monocular estimates carry depth and scale ambiguity. Synchronised, calibrated exocentric views constrain the same frames geometrically, which yields metric labels and a per-clip measure of label error.
Applications
Data format
Delivered as LeRobot, HDF5, or MP4 with JSON annotations.
| Field | Contents |
|---|---|
| video | All exocentric views and the egocentric stream at source resolution and frame rate. |
| timestamps | Per-frame times on a common clock; per-camera offsets recovered in software (no genlock or timecode). |
| calibration | Intrinsics and extrinsics per camera in one metric world frame (z up, ground plane z = 0). |
| body_3d | COCO-17 joints, metres, per frame. |
| hands_3d | 21 keypoints per hand, metres, per frame; per-keypoint view count and reprojection error. |
| error_report | Held-out-camera error: each camera is excluded from the solve in turn and the reprojection residual against its detections is reported in pixels and in millimetres at keypoint depth. |
| robot | Unitree G1 joint trajectory (29 DoF) and physics-tracking metrics: fall, mean joint error, root drift. |
| consent | Per-participant release and commercial licence; optional face blurring. |
Library
| Item | Status |
|---|---|
| current | Outdoor three-drone capture: locomotion, turns, running (Sep 30, 2026). Public sample: 8 s jog with a 90° turn |
| in collection | Workplace tasks with multiple consented subjects: lifting, carrying, shelving, cart pushing, table work, tool use, stairs |
| sources | Opt-in capture sessions only, written consent per participant; pilot data stays with the customer |
| index | Clip list with tasks, subjects, duration and quality grade on request |
Terms
Benchmarks
Harmony4D is used for evaluation only. Its footage is not redistributed or used for training.
| Metric | Result | Protocol |
|---|---|---|
| Software synchronisation | 1.2 ms | Median offset error, p90 2.4 ms. Three cameras, 35 outdoor clips ≥ 4 s, ASPset-510. |
| Body MPJPE | 51.6 mm | Three cameras, outdoor sports, ASPset-510. |
| Hand keypoint error | 17–19 mm | Median held-out-camera error, 3–5 cameras at ~2 m, indoor, Harmony4D. |
| Hand coverage under occlusion | 44% → 88% | Hands triangulated with 3 vs 5 cameras; two-person close contact, Harmony4D. |
| G1 on our drone footage | 60 / 60 | Runs per segment in MuJoCo, 59 with random pushes, friction, mass, motor gains and 0–20 ms delays; three segments from the Sep 30 2026 three-drone flight; joint error 4.2–5.9°. |
| G1, three cameras vs dome | 12.9 vs 12.3 cm | Keypoint error of policies trained on three cameras vs a 27-camera dome reconstruction, tennis clip, 59/59 runs each; single camera adds 3–6 cm. CMU Panoptic, evaluation only. |
| G1, 15-clip study | 551 / 590 | Randomized runs completed by policies trained on three-camera captures across 10 clips (dome-trained: 474 / 590). 5 further sport and dance clips are being re-scored: their dome reference was 0.7 s out of sync with the video. Once aligned, the three-camera capture agrees with it to 5.8–8.0 cm. CMU Panoptic, evaluation only. |
Physics evaluation
Each clip is retargeted to the Unitree G1 with GMR (whole-body IK), checked for start-up transients, unobserved joints and foot contact, then tracked by a motion-tracking policy trained in MuJoCo (mjlab). Evaluation runs 60 times in plain MuJoCo with G1 torque limits: one nominal run and 59 with random pushes, floor friction, link masses, motor gains and control delays. Reported per clip: runs that stay up, keypoint error, bone-direction error and pelvis drift.
| Capture | Runs up | Keypoint error |
|---|---|---|
| Three drones, own flight, 3 segments | 60 / 60 each | joint 4.2–5.9° |
| Three tripod cameras, sprint (ASPset-510) | 58 / 59 | 0.18 m path, median |
| Three cameras, tennis (Panoptic, eval only) | 59 / 59 | 12.9 cm (dome 12.3) |
Simulation only; real-robot runs are next. Write-ups: camera count and robot training, retargeting to a G1.
Pipeline
3–5 exocentric phones at 1.5–2.5 m indoors, 4K, shutter ≤ 1/250 s, locked exposure and focus; one head-mounted camera. Drones for outdoor and moving capture.
Coarse offsets by GCC-PHAT on the audio tracks; refined by maximising epipolar-consistent joint correspondences across views, then jointly to sub-frame precision.
Relative poses initialised by essential-matrix RANSAC on the subject's 2D keypoints, then bundle adjustment of all cameras and points; focal length from lens metadata or estimated; metric scale from body proportions or the subject's height.
Body joints triangulated per frame. Hand crops placed by the 3D wrist; MediaPipe Hand Landmarker per view; per-keypoint RANSAC DLT with a reprojection gate.
Capture envelope
| Parameter | Range |
|---|---|
| exo cameras | 2–5 per session (≥ 3 recommended; 5 for heavy hand occlusion) |
| frame rate | 20–60 fps |
| resolution | 1080p–4K |
| subject distance | ~2 m indoors (hands); up to ~20 m outdoors (body) |
| camera types | Phones, action cameras, consumer drones |
| scenes | Workplace and household tasks, sports, outdoor locomotion; one tracked subject per capture |
| throughput | About 7× clip length on one GPU (a 6 s three-camera 4K clip in 40–45 s); batch processing scales across GPUs |
| processing | Hosted, or on-premises for sensitive sites |