Driving Automata

TL;DR

Driving Automata learns its own discrete driving intents from demonstrations, no labels needed. Enabling Agent/VLM fast but safe request following: Ask it to turn, and it turns. Ask it to run a stop sign, and it won’t.

Abstract

An end-to-end driving policy outputs a trajectory but not its intent, so a passenger cannot tell whether the car will brake or change lanes until it does. Vision-language-action (VLA) models add language, but the planner mostly ignores it, so a passenger cannot use language to change how the car drives.

We present Driving Automata, a driving policy that organizes driving behavior into discrete modes, such as turning, braking, and cruising at different speeds. The policy selects the mode that best fits the current scene, and each mode plans the most suitable trajectory for its behavior from the current observation. Both mode identification and selection are learned from demonstrations without mode labels: a mode-factorized trajectory transformer decoder outputs the trajectories of eight modes, and a sticky hidden Markov model learns the transitions between them and selects the current mode.

We then use the trained policy to label driving data with modes and post-train a vision-language model (VLM) on these labels, which we call mode-grounded VLM training. The VLM explains the current mode with its scene evidence, predicts a possible future mode, and, given a passenger request, selects a mode that both follows the request and keeps the car safe. This gives a single point at which language can be changed and its effect on driving measured.

We evaluate the policy and the VLM in closed-loop simulation with photorealistic 4D Gaussian-splatting rendering (AlpaSim), and introduce SayDo, which tests whether a model's language changes how it drives or only accompanies it.

Driving Automata explanation overlay and mode transition graph — Driving Automata connects driving (the pink trajectory) and language explanation (left). It learns eight semantically distinct modes and their transitions without supervision (right). Each mode handles one driving behavior, which gives the VLM a structured interface to the planner.

Saying is not doing

Generating language and actions together does not guarantee that they are coupled. A model can produce a plausible explanation that does not reflect the decision used by its planner, or acknowledge a passenger request without changing how it drives. We test this directly: we replace the reasoning of Alpamayo-R1 in a stop-sign intersection and run each version in closed loop.

Alpamayo-R1 reasoning intervention in scene 763 — Replacing the reasoning of Alpamayo-R1 in AlpaSim scene 763. Each column runs the scene with the reasoning shown at the top: its own reasoning to yield, turn right, keep straight, a hint that the parking lot is on the right, run the stop sign, and empty reasoning. The model plans the same left turn every time. With its own reasoning, the run ends in a collision.

The corresponding closed-loop Alpamayo-R1 runs in scene 763 (front camera), for the four replaced reasonings, in the order of the figure columns. Each clip starts 1 s before the reasoning is replaced, pauses for 2 s on that frame while the replaced reasoning is shown in the center, and ends 5 s later. Click a video to enlarge.

The problem is structural. Continuous trajectories are precise but low-level, free-form language is expressive but unconstrained, and internal neural features give no stable interface between the two. Driving Automata adds an explicit behavioral variable, the driving mode, that is shared by planning and language and sits on the actual control path.

Method

2.1 Mode-factorized transformer

A frozen DINOv3 encoder turns recent camera frames into scene tokens. A shared transformer decoder attends to them once per latent mode: each of the K = 8 modes has its own trajectory query and an AdaLN mode embedding. Two heads read every mode-conditioned feature: one predicts the mode's 4 s trajectory (8 waypoints at 0.5 s), and one predicts the evidence for that mode.

Mode-factorized transformer and sticky HMM — The transformer answers two questions from the same features: what the candidate trajectories are (one per mode), and what the current mode should be. The sticky HMM turns the evidence into a filtered mode posterior, and the selected mode's trajectory is executed.

2.2 Sticky HMM

Framewise mode choices flicker when two modes score similarly. An online HMM filter propagates the previous belief through a learned, row-stochastic transition matrix with a fixed self-transition bias, then multiplies in the current evidence. A switch happens only when the camera evidence overcomes the propagated belief. The filter is causal and runs with the policy online.

Modes are learned without labels. A winner-takes-all trajectory loss trains only the mode closest to the demonstration; the filtered posterior is trained toward that winner; and a compactness loss keeps each mode a consistent motion pattern across scenes.

p_t^prior(m) = Σ_i p_t−1^post(i) A_im p_t^post(m) ∝ p_t^prior(m) exp(λ e_t,m) A = rowsoftmax(σ(W) + κI)

2.3 Mode-grounded VLM

The frozen policy labels driving data with its own modes. A VLM (the Explainer) is trained on three kinds of data: explaining the policy's current mode, predicting modes from the scene alone, and selecting a mode for explicit, implicit, or unsafe passenger requests. After supervised fine-tuning, GRPO adds a reward from a text-only Listener that must recover the mode from the generated explanation alone, so the explanation has to carry behaviorally discriminative evidence.

Listener-Explainer training — Listener–Explainer training. The frozen Driving Automata supplies mode targets; a text-only Listener rewards explanations from which the policy's mode can be recovered. The driving policy stays frozen throughout.

At inference, a request changes driving only through the mode: the executed mode is the argmax of a mix of the policy's posterior and the VLM's mode distribution, and the trajectory comes from the same decoder. Without a request, Automata keeps control and the VLM explains what it is doing.

Eight modes, discovered without labels

What these 8 self-identified modes look like

The modes separate into braking, left and right turns of two sharpnesses, and straight driving at three speed tiers.

Trajectories per learned mode — Trajectories under each learned mode in the ego frame (x forward, y left, meters), from 30,000 validation windows. Thin: 60 samples; thick: mean. Titles give each mode's share of frames.

Does the transition matrix learn anything?

Every row puts most of its mass on staying in the same mode. The off-diagonals encode which changes the policy expects: braking rarely jumps straight to high-speed driving (0.094 to high-speed cruising, against 0.124 to mid-speed cruising), and turns keep their direction (gentle left drifting moves to sharp left turning more often than to sharp right turning, 0.138 vs 0.104).

Learned transition matrix — Learned transition probabilities (rows: source, columns: destination).

Does the sticky HMM keep the mode stable?

Yes. We vary the filter at inference with the policy frozen, on 5,000 validation clips of 30 planning steps (0.5 s apart). A moderate self-transition bias removes one-step mode flicker and lowers the open-loop L2 error at the same time.

Sticky HMM inference ablation — Changing the sticky HMM at inference. The red line varies the self-transition bias κ (trained value 0.1); the gray and blue lines remove the HMM or its learned transitions. Mode changes: share of steps whose mode differs from the previous step. Flicker: share of steps whose mode lasts one step before the previous mode returns. Mode duration: average time a mode lasts. Open-loop L2: error of the selected plan over the first 2 s. The HMM changes mode less often and flickers less than both ablations; L2 is lowest at κ = 1–2.

Mode identification through turns — Mode identification through right and left turns: gentle lateral motion, a sharp turn, then a return to cruising.

Closed-loop driving

We evaluate in AlpaSim, which renders each next camera frame from the executed ego state with a temporal Gaussian-splatting renderer. Our suite has 131 corner-case scenes: congestion and parking, city and freeway driving, pedestrians, traffic-light transitions, work zones, and constrained road geometry. Values are means over three seeds.

Closed-loop rollouts

Driving Automata alone (no VLM) in 32 AlpaSim scenes, grouped by scenario. Left: front camera with the executed plan (pink); right: bird's-eye view with the eight mode proposals. Click a video to enlarge.

Comparison with other planners

Scene Score vs. total parameters (log scale). Hover a point for details.

Interpretability

Automata drives and the mode-grounded VLM explains it: the current mode with its scene evidence, and a possible future mode.

SayDo: does language change the drive?

SayDo changes the passenger's language while keeping the initial situation fixed, and checks whether the closed-loop behavior changes in the requested way. Ten scenes with more than one plausible behavior are each driven under five conditions: no request, an explicit request, an implicit request that names a goal instead of an action, a different explicit request, and an unsafe request.

1 · Scene

2 · Compare against

Requests followed

Each pair compares two episodes of the same method at the same in-window steps. d_exec: distance between executed ego positions (world frame, m). d_plan: mean distance between plans over the next 3 s (m). Sim.: share of steps at which both plans show the same behavior. A method that follows language separates the episodes: large distances, low similarity.

Refusing unsafe requests

Only Driving Automata can decline a request. The VLM refuses all ten unsafe requests: 97% of its in-window answers start with “ignore request” and name the danger, and no unsafe request adds a collision.

Citation

@article{liu2026drivingautomata,
  title  = {Driving Automata},
  author = {Liu, Hang},
  year   = {2026}
}