Egocentric (head-camera) hand models trained on 3D labels from four outside (exocentric) cameras came within about 1 mm of models trained on ground truth on people they never saw: 24.3 against 23.2 mm wrist-relative MPJPE on HO-Cap. A strong head-camera-only baseline (MediaPipe corrected by a lifter trained on in-domain ground truth) reached 27.5 mm, needed about 1.3 times as much data to match ours at 10% and 1.6 times at 25%, did not match ours at 50% even with all its data, and had 14.5% of hands above 40 mm against our 8.7%. Plain MediaPipe labels from the head camera levelled off at 64 mm however much data was added; ours with 5% of the data beat that. Setup: 8 people in 4 person folds, 3 seeds, 5 data fractions, 5 label conditions (300 training runs).
01The question
A lot of hand data for robot learning is now recorded from head-mounted cameras. The camera is cheap and goes wherever the person works, but it only sees the hands from one point, often half out of frame or blurred when the head turns. The 3D hand labels that come with such footage usually come from a detector such as MediaPipe run on that one view, with depth estimated rather than measured.
We already rebuild hands in 3D from a few outside cameras, to about 5.6 mm on people left out of training (details here). Moved into the head camera’s frame, those 3D hands become labels for the head-camera footage. The question here is how much better a head-camera model gets when it learns from those labels instead of labels from the head camera alone, and how much data each kind of label needs to reach a given accuracy.
02Setup
HO-Cap records 8 people using everyday objects at a table, filmed by a HoloLens head camera and 8 fixed outside cameras, with 3D hand joints for every frame. It is released under CC BY 4.0 by Wang et al. at UT Dallas and NVIDIA. We took every hand with at least 11 of its 21 joints visible in the head camera, on every third frame: 24,359 hands from 64 sequences.
The task is to predict the 21 hand joints from a 224-pixel crop of the head-camera image, as image positions plus 3D positions relative to the wrist in the head camera’s frame. The model is identical in every condition: frozen DINOv2-small features (Apache-2.0) feeding a small two-layer transformer decoder, with the same training schedule and no model selection on labels. Only the training labels change:
- Ground truth: HO-Cap’s own 3D joints, the upper bound.
- Ours: our pipeline run on 4 of the 8 outside cameras (to match a small rig), then moved into the head camera’s frame using the headset’s tracked pose and a calibration step that uses no ground truth.
- Head camera with a ground-truth-trained lifter: MediaPipe on the head camera, corrected to 3D by a lifter trained on other people’s ground truth from the same dataset. This is a generous baseline: a team with only head-camera footage would not normally have in-domain 3D ground truth.
- Head camera only: MediaPipe’s 2D keypoints and relative depth on the head camera, with the wrist depth set so the bone lengths match MediaPipe’s own metric landmarks. No ground truth anywhere. MediaPipe’s 3D world landmarks could not be used directly: on this data they sit about 47 degrees off the camera frame.
The 8 people were split into 4 test folds. Each person is tested exactly once and never appears in training or in any model that produced labels, including our outside-camera pipeline, which was retrained without them. One indirect effect remains: the 3D correction for a training person is fitted on other training people’s tracks, whose 2D refiners had seen that person. That affects how realistic the labels are, not the test people. For each fold we trained on 5, 10, 25, 50 and 100% of the training sequences, with 3 seeds each. All conditions use the same crops, cut around the ground-truth hand box, so hand detection is not part of the test.
03How good are the labels themselves?
| Labels on the training people | Hands labelled | 3D error (median) | 2D error (median) |
|---|---|---|---|
| Head camera only (MediaPipe) | 74% | 53 mm | 36 px |
| Head camera + ground-truth-trained lifter | 74% | 20 mm | 36 px |
| Ours, 4 outside cameras | 93% | 8.6 mm | 12.5 px |
3D error is relative to the wrist, in the head camera’s frame, against HO-Cap’s joints. The 2D figures carry a floor of roughly 30 px from the joint definition (MediaPipe marks the skin, HO-Cap the joint centres) and from the headset’s pose, so compare the 3D column. MediaPipe gave no label for about a quarter of the visible hands; the outside cameras see most of those.
04Results on unseen people
| Training data | Ground truth | Ours (4 outside cameras) | Head camera + GT-trained lifter | Head camera only |
|---|---|---|---|---|
| 5% | 44.5 mm | 44.4 mm | 44.6 mm | 77.0 mm |
| 10% | 33.5 mm | 34.6 mm | 35.9 mm | 65.8 mm |
| 25% | 28.8 mm | 29.7 mm | 31.4 mm | 63.7 mm |
| 50% | 25.7 mm | 26.8 mm | 29.0 mm | 64.6 mm |
| 100% | 23.2 mm | 24.3 mm | 27.5 mm | 64.2 mm |
| Aligned error at 100% | 6.9 mm | 7.1 mm | 8.5 mm | 16.6 mm |
| Hands above 40 mm at 100% | 7.2% | 8.7% | 14.5% | 84% |
Values are the mean over the 8 people and 3 seeds of the wrist-relative joint error. Differences below are paired on the same people and seeds, with 95% intervals from resampling people.
- Against head-camera labels alone, the outside-camera labels were 31 to 40 mm better at every data size, and better for all 8 people. The head-camera-only model levelled off at about 64 mm from 25% of the data on: adding more footage with the same biased labels did not help. Ours trained on 5% (44.4 mm) beat it trained on 100%. That is less a data-efficiency ratio than a ceiling: single-view depth errors in the labels put a floor under the model that more footage did not lower.
- Against the lifter baseline, the two were level at 5% (difference 0.2 mm, interval −1.1 to 1.6). From 10% on, ours was ahead by a growing margin: 1.3 mm at 10%, 1.7 at 25%, 2.2 at 50% and 3.2 mm at 100% (interval 2.4 to 4.0), with all 8 people favouring ours at 50% and 100%. Measured as data, the baseline needed about 1.3 times as much to match ours at 10% (interval 1.06 to 1.57) and 1.6 times at 25% (1.3 to 2.3); with all its data it was still worse than ours at 50%, so about 2 times there, and it never reached ours at 100%. Hands with errors above 40 mm fell from 14.5% to 8.7%.
- Against ground truth, ours stayed about 1.1 mm behind (interval 0.7 to 1.5) from 10% upward, closing about three quarters of the gap between the lifter baseline and ground truth.
Our labels cover more hands (93% against 74%), so part of the gain could come from extra training examples rather than better ones. To separate the two, we also trained on our labels restricted to the crops MediaPipe had labelled (the dotted line). That cost about 0.5 mm. Most of the gain comes from label quality.
05Related work
What 30,000 Hours of Ego-centric Video Does Not Teach (arXiv 2610.12464) reports that a video world model trained on egocentric video reached full agent fidelity at about 300 hours when conditioned on 3D skeleton keypoints, against far more without; the paper calls this 50 times data efficiency. Our experiment is a different and much smaller task: a hand-pose regressor, not a world model, and not a replication. Both say 3D structure makes first-person footage go further; our result is about where such structure can come from, namely a few outside cameras. We are running a world-model version of this comparison with the same label sources and will publish it when it is done.
06Checking head-camera labels, not just replacing them
The same outside-camera hands can audit labels that already exist. We projected our 3D hands into the head camera and flagged MediaPipe labels that disagreed, with the threshold set on the other seven people. On 18,425 head-camera labels from 4 outside cameras, it caught about two thirds of the badly wrong labels (more than 102 px off, 8% of the image width) with about 0.7 precision. Replacing the flagged labels with our projection fixed 74% of all wrong labels. For a quarter of the visible hands MediaPipe gave no label at all; our projection covered 84% of those, at 16 px median error.
This needs a good head pose. HO-Cap’s headset tracks its own pose, and the numbers above use it. On one sequence we also tried our own head-pose estimate: on frames with markers in view or working visual odometry the projection landed about as close as with the headset’s pose, but where the pose came only from the body it was 94–166 px off and the check stopped working.
07What this means for data collection
A head camera on its own records a lot of footage whose labels have a floor that more footage does not lower. Adding a few outside cameras around the work area changes what each session is worth: every frame gets 3D hands measured by geometry, in the head camera’s frame, plus a check on any labels that came from the head camera alone.
That is how we plan to collect: a head camera plus two to four outside cameras, with the outside cameras producing the 3D labels. The same labels also train our own keypoint models, which should let later sessions in similar settings get by with fewer cameras. That loop is our approach, not yet a measured result outside HO-Cap.
08Limits
- The model is small and its backbone is frozen. Trained on ground truth it reaches 23 mm, and that floor compresses the differences between labels: 8.6 mm against 20 mm labels showed up as 24.3 against 27.5 mm. A fine-tuned or larger model would probably widen the gap; we have not tested that.
- The lifter baseline is generous on purpose. It learned from in-domain ground truth from the same headset. A lifter trained on another dataset would likely land between the two head-camera rows.
- HO-Cap is small and lab-based: 8 people at a table, slow head motion, one headset. The 5% points use 2–3 sequences each and vary a lot.
- All conditions use crops around the ground-truth hand box. Hand detection, and how each label source copes with hard frames, is only partly tested.
- Our labels used HO-Cap’s calibrated and synchronised outside cameras and the headset’s tracked pose. Our own capture pipeline, tested on a HO-Cap sequence set up like our rig (three outside cameras, calibration from markers, no shared clock), reaches about 8.5 mm on an unseen person, so labels from them should be slightly worse than here. Without a tracked head pose, moving the labels into the head camera’s frame is much less accurate.
- The head-camera-only row uses MediaPipe’s own depth. We have not tested a stronger single-view 3D hand model such as HaMeR or WiLoR; the lifter row is our proxy for the best a head-camera-only team could do, and that gap is 1.3–2 times in data, not 20.
- One dataset. A second, larger ego-exo dataset is the next check.
Questions
How accurate are MediaPipe hand labels for egocentric data?
On HO-Cap’s head camera, MediaPipe 2D keypoints with its own relative depth gave 3D hand labels with a median error of 53 mm (wrist-relative), and labelled 74% of the visible hands. A model trained on those labels stalled at about 64 mm on unseen people, however much data it got. A lifter trained on in-domain ground truth brought the labels to 20 mm.
Do 3D labels from external cameras improve egocentric hand models?
In our HO-Cap test, yes. The same head-camera model trained on labels from 4 outside cameras reached 24.3 mm on unseen people, against 23.2 mm with ground truth, 27.5 mm with ground-truth-corrected head-camera labels and 64.2 mm with head-camera MediaPipe labels alone. Trained on 5% of the data, it beat the MediaPipe-only model trained on all of it.
Can outside cameras check Aria or HoloLens hand labels?
Yes, if the headset’s pose is known. With 4 outside cameras and HO-Cap’s tracked HoloLens pose, projecting our 3D hands into the head view caught about two thirds of badly wrong head-camera labels at about 70% precision, and replacing the flagged labels fixed 74% of wrong labels. Without a tracked pose or markers in view, the check is much weaker.
How many outside cameras do you need?
These results used 4 outside cameras, which gave labels with an 8.6 mm median error in the head camera’s frame. Our own capture pipeline reached about 8.5 mm with 3 cameras on an unseen person, in a test set up like our rig. You also need the head camera’s pose, from the headset’s own tracking or from markers in view.