How do we plan in complex environments?

Human behavior, from infants to adults, is shaped by temporal regularity in sequences such as speech, music, or motor movements. We are sensitive to the transition probabilities between syllables, notes, or other items, and can use those probabilities to guide our perceptions and expectations of the immmediate future. But to plan further into the future, we either need to chain these single-step predictions (which becomes increasingly costly with planning depth) or somehow generate multi-step predictions that themselves reach beyond the next event

We ask how people extract that higher-level structure, starting from the mechanisms that support simple statistical regularities and asking how they can underpin multi-step temporal abstractions. We study this by studying how people learn to predict sequences of items. Formalizing such sequences as walks on a graph, encoding the possible transitions between items, allows us to hold local statistics fixed while varying the global structure that generates them, and examine how people depend on such global structure. We can then read out peoples’ internal predictions via reaction times and choices, and examine how these predictive models are formed trial-by-trial.

The brain is capable of both habitual and goal-directed behavior, typically mapped onto model-free planning, which caches the long-run values of actions, and model-based planning, which simulates outcomes through a world model, and these two strategies are though to adaptively trade off with one another. Planning in complex environments, however, requires predictions over temporally extended timescales, where the value of an action may depend on outcomes many minutes away. In such environments, habits are insensitive to changes in distant outcomes, while simulation can be too expensive for longer horizons.

Predictive models such as the successor representation offer a middle path, learning where actions lead separately from what those outcomes are worth, so distant values can change in the face of new information, without costly step-by-step replanning. Substantial evidence demonstrates that people rely on such predictive models in deicison making, and we have found that such models trade off against finer-grained model-based planning. In particular, the brain adaptively chooses finer-grained models only when coarser predictive models are behaviorally insufficient. However, understanding what signals the brain relies for this arbitration remains an ongoing question.