LATTICE Robotics & physical AI

Grounding robot intelligence in the physical world

Give robots a physical model of the world.

Lattice turns a robot's workspace into a live 4D model: the robot, the people and the objects around them, all tracked as they move. Teams use that model to build realistic simulations, train robots, and check how they perform in the real world.

SeeRobots, people and objects in one metric 3D scene, accurate in time, with egocentric views from the people and robots in it.
GroundTurn real geometry and motion into simulation environments and data, then measure where simulated behavior diverges from reality.
LearnTrain and evaluate policies on real 3D world state and real human motion, then measure what happens when the robot acts.
Building the workcell capture

IllustrationNot real capture or reconstruction data

A robot workcell Drag to scrub · arrows step

Robots need context, not just pixels.

  • A camera frame shows color. A task needs context: where things are in 3D, who is nearby, what just moved, what is hidden behind the arm.
  • Lattice fuses synchronized views into one 3D scene, tracks bodies and hands joint by joint with no mocap suit, and renders first-person views from the same capture.
  • The difference shows up in results. beat hand-built ones on 5 of 5 captures, and in our (simulation, research in progress), planning on a 3D world state rather than the current frame raised task success from 0 of 60 trials to 24 of 60.

Make simulation answer to reality.

  1. Real
  2. Measure
  3. Simulate
  4. Train
  5. Act
  6. Compare

Lattice turns real scenes and interactions into measured simulation twins. Policies train and act against them, and the sim-to-real gap stops being a guess: it is measured joint by joint and fed into the next iteration.

Sim predicts. Lattice observes. The residual corrects the sim.

This is the system we are building. Our benchmarks test each link on its own, and every result is registered in the ledger.

Reality, delivered as data.

METRIC 4D STATESYNC RGB + DEPTHCALIBRATION + TIMESTAMPSOBJECT + HUMAN MOTIONEGOCENTRIC VIEWSROBOT TRAJECTORIESSCENE STATEREPLAYABLE EPISODESSIMULATOR HANDOFFREAL / SIM RESIDUALS

The cameras are the measuring instrument. The product is the physical state they record, in one metric frame on one accurate clock, built to hand off to your simulator and training stack.

Ground every policy in the world it will actually run in.

Measure the physical world, build simulation from it, train and test against it, then measure what really happens on the robot. Each pass can close more of the gap between what the policy expects and what the world does.

Benchmarks

Does better physical state change simulation and robot behaviour?

LEDGER 2026-10-02

Each result is a registered entry in our benchmark ledger. Every card names what was changed, what was held fixed, the sample, and the caveat that travels with the number.

Primary

01

Measured scenes beat hand-built scenes on every capture.

Reality to simulation

5/5

captures where the measured surface matched the observation better, in both directions

Does a simulation surface built from measured geometry match what the cameras saw better than one an engineer authors by hand?

Treatment
Surface built from the Lattice reconstruction.
Control
Hand-authored surface, deliberately strengthened: placed on the best-fit plane of the observed cloth at the same 3 mm lattice.
Sample
5 captures, 2 object types (bags, shirts). Lattice is 1.5–2.4× closer on every capture.

CaveatInitial geometry only. No simulator ran in this benchmark, so it does not evaluate simulated dynamics.

Method and full caveats
  • Metric: median distance between surface and observation, measured both ways through each capture’s own calibration. Preregistered; all seven gates passed.
  • Identity control reads 0.000 mm on all five captures; a 10 mm displacement reads 7.59–8.40 mm.
  • The reverse direction (observation to surface) penalises a sparser surface for being sparse; Lattice’s 1.529–1.660 mm clears its own 1.950–2.101 mm resampling floor, so the comparison holds, but that factor should not be over-read.
  • The factor is smaller than earlier single-capture work because the hand-built arm here is stronger. The consistency across five captures is the result, not any single factor.

Ledger led-2026-09-10-0292

02

A measured material makes the cloth twin follow reality more closely.

Physical parameters to better simulation

8.84% → 7.01%

whole-take divergence of the simulated cloth from the real take, hand-picked vs measured material

If the cloth’s bending stiffness is measured rather than picked by taste, does the simulated twin track the real take better?

Treatment
Measured bend stiffness (10,000).
Control
A hand-picked value (100). Same policy, seed, 1,200 steps and five grips in both arms.
Sample
6 runs per arm on one real take; the gap is 14.7 standard errors. Skill against doing nothing rises 0.725 → 0.783.

CaveatOne take. Over the fold segment alone the hand-picked value is better (4.34% vs 4.85%), so this supports the whole-take objective only.

Method and full caveats
  • Whole-window divergence 8.837 ± 0.269% (hand-picked) vs 7.007 ± 0.145% (measured).
  • Every run carries a numeric readback of its requested material; a first attempt with a formatting bug in that readback was discarded.

Source reconstruction/Measurements-2026-09-11-Cloth-Material-AB

03

Small reconstruction errors do not stay small.

Reconstruction to dynamics

3.43×

median amplification of a reconstruction difference after rollout, against a preregistered 2.0× bar

With policy, material, seed and solver held fixed, does a small difference in the reconstructed scene stay small once the simulator runs?

Varied
Only the reconstructed cloth: raw four-camera and single-camera meshes against the Lattice mesh. Camera 0’s 7.50 mm difference became 25.72 mm of rollout divergence.
Mechanism
Coverage produced 1.96× more absolute divergence than precision (2.44× inside the preregistered window), while precision had the larger amplification factor, 7.54× vs 3.84×, from a 9× smaller base. Both halves belong together.
Sample
33 rollouts in one interleaved batch; the reference arm was repeated 5 times for the noise floor.

CaveatSim to sim. It shows that reconstruction changes what the twin predicts, not which prediction is closer to the real robot. One take, one cloth, one batch.

Method and full caveats
  • Camera 2’s 1.26× is not a statement that camera 2 is good: it carries a known 123.8 mm defect, and the finding is that the simulator is insensitive to that specific error.
  • The noise floor is bimodal: past about 16 s every figure is noise-dominated. A whole-episode offset gate was refused, and the bar was not moved.
  • Cameras 1 and 3 were excluded by a preregistered coverage bar. One room, one rig, one operator, one cloth.

Ledger led-2026-09-17-0415

Supporting

04

Single view vs multi-view hand tracking.

Multi-view for manipulation

1.72×

higher hand-joint error from a single camera than from multi-view fusion, scored on a camera neither saw

How much hand-tracking accuracy does a single RGBD camera, the common robot setup, lose against multi-view fusion?

Treatment
Multi-view fusion from the rig, with one of the four cameras held out to score every arm.
Control
One RGBD camera lifting its own pixels along its own rays.
Metric
Hand-joint reprojection error into the held-out camera, paired per frame and keypoint.
Sample
Single view was worse on 13 of 13 captures that passed both preregistered gates.

CaveatPixels, not millimetres. One operator, one rig. Most of the gain comes from the second view.

Method and full caveats
  • The gates refused 10 of 28 takes (6 on a 17 mm evidence floor, 4 on a preregistered wrong-fingertip control), including the corpus’s single highest ratio.
  • Seven captures regenerated by two independent pipelines agree to 0.030 median, 0.049 worst case.
  • This replaces an earlier 2.2× headline that came from the top two captures. Quote 1.72× with its range.
  • Whether a model trained on these hand motions is better is not yet tested.

Ledger led-2026-09-07-0163, superseding 0162

05

Depth at a camera the rig never saw.

Multi-view coverage

2.94×

lower depth error than the strongest multi-view model where both predicted, while covering 61.4% of the view against its 9.8%

Given three of the four cameras, how well can each method predict depth at the fourth, and how much of that view can it predict at all?

Treatment
Calibrated multi-view capture from three cameras, with the fourth held out as the evaluator.
Control
External depth models given the same three views (MapAnything with true intrinsics, UniDepthV2, MoGe-2, VGGT, Depth Anything 3), and a single camera.
Sample
5 takes × 8 frames × 4 held-out cameras: 160 cells per arm, 0 refused. An identity control reads exactly 0.

On the identical pixels every arm covers, the error is 14× lower than UniDepthV2 and MoGe-2.

CaveatRead both columns: the gain is mostly coverage. One camera to three raises coverage 29.3% to 61.4% (2.10×) but accuracy only 1.12×. Our refined output is indistinguishable from raw multi-view fusion here, so this credits calibrated multi-view capture, not the refinement layer.

Method and full caveats
  • A single calibrated camera scores 6.56 at 29.3% coverage; a constant (no information) scores 135.56.
  • Coverage is not the highest: UniDepthV2 and MoGe-2 cover about 64.7%, at roughly 10× the error.
  • Refined output vs raw fusion is a replicated null across five takes (p = 0.625 on this data), so no win is credited to refinement.
  • One room, one rig, four cameras of one model; no cross-room claim.

Ledger led-2026-09-11-0314, 0327

Research in progress

06

Remembered scene state changes what a robot can attempt.

Persistent state to policy

24/60 vs 0/60

trials of a four-step cable task completed in simulation, all with operator handovers

24 / 60Persistent 3D world state

0 / 60Current frame only

Can a robot that keeps a persistent 3D model of its scene finish a task that the same robot acting only on its current frame cannot?

Task
Plug, plug, unplug, plug on a simulated Dexmate Vega, across 60 random placements (seed 20260927).
Treatment
A scripted task runner planning against a persistent 3D world state, where unknown space is never treated as free.
Control
The same runner and placements, acting only on the current frame. It refused 55 of 60 at the base path because it no longer held what its own survey had seen.
Sample
60 placements per arm; 0 unsafe strokes in either arm.

CaveatResearch in progress. Simulated world, no physics stepped, nothing ran on a robot. Every success needed operator handovers; with none, the persistent arm also completed 0/60. The control run’s transit route differed (279 vs 316 rows), so the registered same-motion criterion is not strictly met.

Method and full caveats
  • Preregistered as PREREG-vega-world-state-ablation-20260930.
  • Handovers are an operator stand-in; robot hand-to-hand transfers are not in the runner. The 0.21 m park band is an operator observation, not sensed.
  • Persistent-arm refusals: 15 at the base stance, 16 slides refused because unseen under-table space now counts as table, 5 other.

Ledger led-2026-09-30-0552

Not shown as results: the cloth twin against the best single camera (provisional; it carries known twin defects and will be re-run after the cloth rate-invariance work), the 3D-data ablation on that twin (a tie within its 7.7 mm noise floor at n = 6), and refined versus raw multi-view fusion (a replicated null). The full records are available on request.

Every figure traces to a registered ledger entry Ask for the full ledger