Robotics & Embodied AI · entry 02/05

Sense, plan, act

Perception, state estimation, planning, control — the layered architecture that runs most working robots, and the end-to-end policies now blurring every boundary in it.

The stack, top to bottom

Classical robotics is a bureaucracy of well-defined jobs. Perception turns sensors into a world model: detect and segment the objects, estimate their poses. State estimation answers "where am I" — SLAM builds the map and localizes in it simultaneously, fusing wheel odometry, IMU, and vision into one best guess. Planning finds a collision-free path through configuration space and sequences the task. Control turns plans into torques, in loops running hundreds to thousands of times per second. Each layer has its own clock rate, its own math, and its own failure alarms — and that legibility is why the stack still runs warehouses, surgical suites, and Mars rovers.

The seams are where it breaks

Layered designs fail at the interfaces. Perception commits to "the handle is here" and planning treats the guess as fact; uncertainty gets rounded away at each handoff. The plan that was optimal at t=0 meets a world that moved by t=2. Contact-rich moments — insertion, wiping, anything deformable — resist the tidy geometry planning wants. Engineers patch the seams with replanning rates and error recovery states, and every patch adds a mode, and every mode multiplies the testing surface.

End-to-end learning's bet

The learning approach collapses layers: camera pixels in, motor commands out, one network trained on demonstrations or experience. No pose estimates, no explicit map — whatever intermediate representation helps, the network invents. The gains are real: learned policies handle the fuzzy, contact-rich tasks the stack fumbles, interpolating through the middle ground where symbolic layers snap — though off-distribution they fail silently rather than gracefully. The costs are equally real: no layer to inspect when it misbehaves, no guarantee to certify, and data hunger in a field with no data.

What ships is hybrid

Production systems increasingly split the difference: learned perception and learned skills inside a classical skeleton — a planner that sequences neural policies, safety monitors with veto power over any of it, hard-coded envelopes on speed and force. The stack becomes scaffolding for learning rather than its rival. It's less pure than either camp's pitch deck, and it's what actually survives contact with a factory floor.

Failure mode

Trusting confidence at the interfaces. The classic incident: the perception module reports the pallet at 94% confidence, the planner treats it as certain, the arm commits, and the 6% happens at full speed. Whatever the architecture, propagate uncertainty to the layer that can act on it — slow down when unsure, look again, widen the margin — because a robot that knows when it doesn't know is worth more than one that's usually right.