A robotics team could fit a model to logs of camera observations and steering actions. Given a candidate turn, the model predicts the next observation or a compact representation of it. Rolling several actions forward lets a planner compare possible routes before trying them in the physical environment.
Ha and Schmidhuber’s World Models paper demonstrates learning compact visual and temporal representations and training a controller in a learned environment. A hand-written simulator can also support model-based planning, but its dynamics are specified rather than learned from data. Many digital twins work this way: an engineered model of one machine or site, kept in sync with its live sensor data.
Terminology varies: a product called a world model may predict video without exposing an action-controlled planning interface. Ask what it predicts and how actions enter the model. Predictions can accumulate errors over a rollout, so an attractive imagined route is not evidence that a robot can execute it safely. Check plans against the real task. Spatial intelligence and physical AI cover the perception and action problems these predictions feed into.
Sources
- Ha and Schmidhuber: World Models — Learns compact spatial and temporal representations and trains policies inside a learned world model.
Go deeper
- Hafner et al.: Mastering Diverse Domains through World Models paper
Explore Dreamer’s use of learned dynamics and imagined trajectories for policy learning.