A robot sim or a footstep planner needs the ground where the robot will actually stand. We tested how well ordinary cameras give it that, at real footfalls of a real robot, and then whether the difference shows up in what a walking policy does.
In short. On natural ground, 90% of ANYmal-D footfalls land within 8.6 cm of our camera terrain and 70% within 5 cm. The dataset’s lidar map does better, 6.3 cm, and guessed flat ground and single-camera depth do far worse (34 and 55 cm). Lidar wins on almost every terrain type. On the walking side, ETH’s G1 policy steps off our measured table successfully 91% of the time, against 66% when its map is ground guessed from the motion. A simulation built on the guessed ground reports 100%.
Why footfalls
Terrain models are usually scored against another sensor, one scan against another. A walking robot offers an independent check. During every stance, the joint encoders and the robot’s kinematics say where the sole is, and that point should sit on the ground the robot is standing on. We used about 31,000 footfalls from 17 missions of GrandTour, ETH Zurich’s open dataset of an ANYmal-D walking through forests, alpine gravel, snow, meadows, towns, an industrial site and a construction site. For each footfall we measured the height gap between the foot and each terrain model under it.
Our terrain comes from the robot’s own cameras and nothing else: no lidar and no footfall data go into it. It is compared with the dataset’s lidar map, with flat ground fitted to the robot’s path (what a simulator uses when nobody scanned the site), and with depth from a single camera.
Natural ground
On forest floor, leaf-covered forest stairs, alpine gravel and snow, which make up 43% of the footfalls, the worst 10% of footfalls sit 8.6 cm or more from our camera terrain (95% CI 8.2–8.9). The lidar map scores 6.3 cm (6.0–6.6). Flat ground fitted to the path scores 34.2 cm, and single-camera depth 55.1 cm. Within 5 cm: 70% of footfalls for ours, 83% for lidar.
Pooled over every footfall on every mission, ours is at 16.6 cm and lidar at 9.8 cm, with flat ground at 33.4 cm and single-camera depth at 45.1 cm. Most of the gap between ours and lidar comes from a few conditions, covered below.
A construction site
Mission CON-4 is a loop around a residential construction site: wet concrete slabs, compacted dirt, tyre tracks, debris and a dirt mound, with lorries and workers moving through. Over the whole mission, 90% of footfalls sit within 6.3 cm of our camera terrain and 81% within 5 cm. Lidar does better again, 3.2 cm and 97%. Flat ground scores 11.9 cm and single-camera depth 11.6 cm.
The site is mostly flat, which flatters the flat-ground guess. It would not see a trench, a kerb or a pile of material before the robot reached it. The one bad stretch for the cameras is the first 80 seconds, on wet, reflective concrete with lorries parked alongside: our error reaches 18 cm there while lidar stays near 2 cm.
Terrain by terrain
Each 10-second interval of every mission was labelled with a terrain type from the robot’s front camera. The lidar map is more accurate than our cameras on every type except urban and metal stairs, where both are poor at about 21 cm. On leaf-covered forest stairs the two are close.
Forest stairs, leaf-covered
Forest floor
Pavement
Short grass
Tall grass and undergrowth
Dark indoor floor| Terrain | Share of footfalls | Ours, worst 10% | Ours, within 5 cm | Lidar, worst 10% | Lidar, within 5 cm | Guessed flat ground |
|---|---|---|---|---|---|---|
| Forest stairs, leaf-covered | 1% | 8.3 cm 6.6–8.8 | 72% | 7.1 cm 5.9–8.0 | 77% | 27.5 cm |
| Forest floor | 27% | 8.5 cm 8.0–8.9 | 70% | 6.8 cm 6.5–7.1 | 80% | 31.5 cm |
| Alpine gravel and boulders | 8% | 9.6 cm 8.4–11.2 | 74% | 4.5 cm 4.1–4.8 | 92% | 61.0 cm |
| Snow | 6% | 8.4 cm 7.8–9.0 | 68% | 5.7 cm 5.4–6.2 | 85% | 25.3 cm |
| Grass, short and tall | 14% | 32.3 cm 30.9–33.8 | 14% | 21.3 cm 20.1–22.6 | 20% | 22.2 cm |
| Pavement and concrete | 30% | 10.6 cm 9.2–12.2 | 80% | 2.9 cm 2.8–3.0 | 96% | 27.7 cm |
| Urban and metal stairs | 5% | 20.5 cm 17.0–25.3 | 51% | 20.9 cm 15.3–29.7 | 67% | 72.1 cm |
| Construction-site dirt | 2% | 6.2 cm 5.7–6.9 | 75% | 4.1 cm 3.9–4.5 | 95% | 8.8 cm |
| All terrain | 100% | 16.6 cm 15.6–17.8 | 65% | 9.8 cm 9.0–10.5 | 78% | 33.4 cm |
Worst 10% is the 90th percentile of the height gap between foot and surface, with its 95% confidence interval. Guessed flat ground is one plane per 80-second stretch through the robot’s path.
Knowing when it is wrong
Our reconstruction marks each patch of ground as confident or not, from the images alone, before any footfall is seen. Keeping only the confident patches covers 73% of footfalls and brings the pooled worst-10% error from 16.6 cm to 11.5 cm. On pavement it goes from 10.6 cm to 5.3 cm at 69% coverage. A user can keep those patches and fill the rest from another source. It does not help in tall grass, where the cameras are confidently measuring the tops of the grass.
Feet sink below the surface
Every terrain model here puts the ground slightly too high. On pavement, leaf litter and snow, the foot comes to rest about 1.5 to 3.5 cm below the measured surface, for lidar and cameras alike. About 1.5 cm of that shows up even on pavement and comes from the foot pad and where the contact point sits on the foot; the rest is material the foot pushes through. In tall grass and undergrowth the foot ends up 13 to 21 cm below the visible surface, because both sensors see the tops of the grass. There, flat ground (9.9 cm) beats both lidar (24.9 cm) and our cameras (36.3 cm).
| Terrain | Foot vs our surface | Foot vs lidar surface |
|---|---|---|
| Pavement | −2.4 cm | −1.6 cm |
| Alpine gravel | −2.7 cm | −1.5 cm |
| Snow | −3.3 cm | −2.1 cm |
| Forest floor, leaf litter | −3.5 cm | −2.8 cm |
| Construction-site dirt | −4.1 cm | −2.3 cm |
| Short grass | −8.5 cm | −7.0 cm |
| Tall grass and undergrowth | −20.9 cm | −12.8 cm |
The gap is systematic enough to correct. We learned an offset per terrain type on the other missions and applied it to each held-out mission. Our pooled error drops to 12.3 cm, and lidar’s to 6.8 cm.
| Correction for the gap, learned on the other missions | Ours, worst 10% | Lidar, worst 10% |
|---|---|---|
| None | 16.6 cm | 9.8 cm |
| One offset for all terrain | 14.2 cm | 8.4 cm |
| One offset per terrain type | 12.3 cm | 6.8 cm |
Tall versus short grass is a split we added after seeing the results; it was not in the pre-registered terrain types.
Where the cameras fail
Most of the gap to lidar comes from four conditions:
- Sunny, low-texture ground with the robot’s own shadow moving over it, such as asphalt in a town (about 24 cm against lidar’s 3 cm), and wet, reflective concrete on the construction site.
- Metal grating, whose repeating pattern fools the cameras into a surface tens of centimetres off.
- Dark indoor floors at the industrial site, where lidar also shows errors of about 10 cm.
- Tall grass and undergrowth, where both sensors see the canopy.
Open metal stairs defeat a single height-per-point map for both sensors. We also tried estimating the robot’s position from the cameras alone, with no lidar anywhere. It matched on one forest stretch and drifted on featureless snow and on sunny town stairs, so every number on this page uses the dataset’s lidar-inertial positions for the robot, shared by all terrain models.
Does it change the policy?
A few centimetres of terrain error only matter if a robot acts on them. Watch Your Step, from ETH Zurich’s Robotic Systems Lab, is an open walking policy for the Unitree G1 that reads a local elevation map and picks its footholds from it. We ran the released checkpoint, unmodified, on three sites from Capture 03: a picnic table, a hill descent and a levee crossing, each following a route a person walked. Every site was run three ways:
| Measured terrain | Physics and the policy’s map both use our measured ground. |
|---|---|
| Guessed map | The same measured physics, but the policy’s map is ground guessed from the person’s foot contacts, with no objects. Only what the robot believes changes. |
| Guessed-ground sim | Physics and map both use the guessed ground: what a simulation built without a scan would report. |
On the clip where the robot walks along the table and steps down, it succeeds 91% of the time on the measured terrain (95% CI 88–95%) and 66% with the guessed map (60–72%), a paired difference of 25 points (19.9 to 30.5). It falls in 9% of episodes on the measured terrain and 34% with the guessed map. A simulation built on the guessed ground says 100%. Over the whole table site the policy succeeds 46% against 33% (44–47% and 30–36%).
The table and the slopes
The guess turns the narrow table, about 0.55 m wide, into a ramp. With the guessed map the policy steps off an edge it cannot see: starts within 10 cm of the edge succeed 74% of the time on the measured terrain and 7% with the guessed map. On the table site, 35% of the map the policy reads is more than 5 cm wrong.
On the hill descent and the levee crossing the guess is good to 1–2 cm at the footholds (median; worst 10%: 5.9 cm on the hill, 3.6 cm on the levee). That is inside the noise the policy was trained with, and the policy succeeds 100% of the time in every condition. The guessed-ground simulation is wrong in both directions: on the step-down it reports success where the measured terrain gives 91%, and on the 0.6 m step-up it reports a fall every time where the measured terrain gives 83%. That step-up is higher than anything the policy was trained on (stairs up to 0.25 m, blocks up to 0.3 m), so it fails on every ground.
Checked against the paper
Before running our sites we ran the checkpoint on ETH’s own test terrains to make sure our setup reproduces the policy. It walks as well or better than the paper reports on falls and corridor goals, with somewhat lower velocity tracking.
| On ETH’s own test terrains | Our run | Paper, Table II |
|---|---|---|
| Velocity tracking, training / unseen terrains | 75.9% / 74.0% | 84.6% / 83.1% |
| Falls, training / unseen terrains | 0.1% / 1.0% | 3.3% / 5.2% |
| Corridor goal reached, all corridors | 96.8% | 89.2% |
| Foothold prediction error, first / second step | 3.6 / 7.2 cm | about 6 cm per step |
Protocol
| Footfalls | Stances of at least 0.12 s from the robot’s contact state. The foot point comes from the 12 joint encoders through the ANYmal-D kinematics. On flat pavement, footfall heights fit a plane to a median of 2.5 mm |
|---|---|
| Metric | Foot height minus surface height at each footfall. Reported: the worst-10% gap (90th percentile), the share within 5 cm and the median offset; every method scored on the same footfalls, each with its coverage (lidar 99%, ours 92%, single-camera depth 97%) |
| Baselines | The dataset’s lidar map; one flat plane per 80 s through the robot’s path; single-camera metric depth (MoGe-2), rescaled per frame to the robot’s leg height, which is generous to it |
| Statistics, Part 1 | 95% confidence intervals from a block bootstrap over 2-second blocks, so all four feet and neighbouring steps resample together |
| Policy runs | 256 episodes per clip per condition from identical paired starts (route start ±0.2 m, heading ±30°). Success: reach the end of the route and stand on the end surface, with no fall |
| Statistics, Part 2 | Stratified bootstrap with 10,000 resamples; differences between conditions are paired |
| Pre-registration | Both studies fixed their metrics, conditions and episode counts before the main runs. Part 1’s settings were tuned on one 80-second development stretch (forest stairs), the only one with results at the time. Part 2 had one amendment, made before any result: “standing on the end surface” was added to the success test |
Credits
- GrandTour: ETH Zurich Robotic Systems Lab; Frey, Tuna, Fu et al. (arXiv 2602.18164); dataset on Hugging Face, MIT licence. Robot: ANYmal-D, kinematics from ANYbotics’ description (BSD-3-Clause).
- Watch Your Step: Heyrman, Li, Dadiotis and Hutter, Robotic Systems Lab and ETH AI Center, ETH Zurich (arXiv 2610.29612); code and checkpoint, MIT licence. Run unmodified; the measured-versus-guessed comparison is ours.
- Sites: the picnic table, hill and levee from Capture 03, October 17, 2026. Physics: Isaac Sim and Isaac Lab. Robot: Unitree G1.
Numbers on this page are generated from the result files at build time. The terrain under the earlier terrain training study and the terrain lab comes from the same capture.