Behavior Prompting Policy: demonstrations as prompts for robot control

21 September 2026The Robotensor team5 min read

Behavior Prompting Policy (BPP) lets robots learn new tasks from a single demonstration at inference time, using the demo as context rather than retraining the model. Its strongest results suggest that behavioral diversity matters more than repeated demonstrations, enabling robots to recombine familiar skills into new tasks without gradient updates.

What if teaching a robot a new task looked less like retraining a model—and more like simply showing it once?

Behavior Prompting Policy (BPP) explores exactly that.

A breakdown of how it works, what actually matters when training it, and where its limits are 🧵👇

architecture

1/ Behavior prompting = in-context learning for robots

Give the robot:

one demonstration of the task — the behavior prompt
its current observation

BPP then generates closed-loop robot actions without test-time fine-tuning.

Instead of only specifying what the final goal should look like, the demonstration also shows how to get there.

2/ What is inside a behavior prompt?

The prompt contains:

observations + proprioception + actions

Processing every visual frame would make long demonstrations expensive, so BPP compresses them:

observations/proprioception → typically ~1 Hz
actions → kept at full temporal resolution
each segment → compressed into one prompt embedding

So the model gets a relatively short representation while preserving detailed motion.

3/ The prompt encoder

At every step, the model effectively asks:

“Given what I see now, which part of the demonstration matters?”

The current observation cross-attends to the demonstrated trajectory.

This is important because the demonstration and the current rollout do not need identical object positions or timing.

Published prompt-encoder config:

6 Transformer layers
hidden dim: 768
MLP: 3072
8 attention heads
~57M parameters

4/ From prompt → robot actions

After retrieving relevant information from the demonstration:

**current observation

relevant prompt information → diffusion action decoder → action chunk**

Published decoder config:

CNN U-Net diffusion policy
diffusion embedding: 128
channels: [256, 512, 1024]
kernel size: 5
~151M parameters

An important design choice is separating prompt understanding from action generation.

The whole demonstration doesn't need to be repeatedly reprocessed during every diffusion denoising step.

5/ How BPP actually learns this

Suppose we have multiple demonstrations of task X.

Training can look like:

Demo A → behavior prompt

state from Demo B → current observation

future actions from Demo B → prediction target

Both demonstrations describe the same task, but their timing, object positions, and environment state may differ.

There is no manually provided temporal/spatial alignment.

The network learns that correspondence end-to-end.

6/ One of the most interesting findings: diversity > repetition

For the DrawAnything-Sim ablation, with roughly the same total demonstration budget:

more tasks × fewer demos/task

performed better on unseen-task prompting than:

fewer tasks × more demos/task

The main DrawAnything-Sim setup used:

2,000 tasks × 5 demonstrations/task

So for behavior prompting, the number of distinct behaviors the policy experiences may matter more than repeatedly demonstrating a smaller task set.

7/ This changes how I'd think about tuning BPP

For standard imitation learning, it's natural to ask:

“How many demos do I have per task?”

For BPP, another question may be even more important:

“How many different behaviors does the model understand?”

Dataset composition becomes a major hyperparameter.

More demonstrations are not automatically more useful if they come at the expense of behavioral diversity.

8/ The paper also shows the other side of that result

Their laundry setup only had 3 training tasks.

There, behavior prompting was less reliable at identifying the intended behavior, while explicit language conditioning could provide a clearer task signal.

So BPP's prompting ability shouldn't be treated as something that simply appears from the architecture.

It appears strongly tied to the diversity of the training distribution.

9/ Training scale

For reference, the reported experiments used 4 GPUs, with training taking roughly 24–42 hours depending on the benchmark.

Examples:

DrawAnything-Sim → ~27 h
DrawAnything-Real → ~42 h
LIBERO-Gen Chain → ~34 h
Laundry → ~24 h

These aren't universal BPP requirements—but they're useful scale references.

10/ Does the prompt actually matter?

On unseen DrawAnything-Sim drawings, BPP reports:

80.7% lower error than Goal-Image conditioning
33.3% lower error than ICRT

What's particularly interesting is the attention behavior.

As execution progresses, attention moves through corresponding regions of the demonstrated trajectory.

One interpretation from the paper:

the demonstration acts like a sequence of dense intermediate sub-goals, rather than a single final goal.

11/ BPP vs ICRT

This makes the comparison with ICRT especially interesting.

ICRT also explores in-context imitation from sensorimotor demonstrations—but formulates control as autoregressive next-token prediction.

BPP instead combines:

compressed behavior prompts + cross-attention + diffusion action generation

Two different ways of asking the same larger question:

Can robot policies learn to treat demonstrations as context rather than training data?

cc: @letian_fu @Ken_Goldberg

12/ And LIBERO matters here too

The paper introduces LIBERO-Gen to evaluate unseen combinations/sequences of tabletop manipulation behaviors, building on the LIBERO ecosystem.

But there's an important nuance:

“new task from one demo” does not mean the robot sees a completely unknown manipulation primitive once and instantly masters it.

The paper explicitly notes that BPP does not yet establish one-shot acquisition of entirely unseen tabletop action primitives.

The stronger supported claim is:

BPP can use a demonstration to specify and adapt execution at inference time by generalizing/recombining behaviors represented in a sufficiently diverse training distribution.

cc: @yifengzhu_ut @yukez

13/ The bigger idea

BPP is interesting beyond this particular architecture.

Today, robot demonstrations are usually treated as:

data used to train a policy

Behavior prompting asks us to also treat them as:

context used to program a policy

That's a meaningful shift.

Language models gave us:

prompt → new behavior

Robot learning may increasingly move toward:

demonstration → new behavior

No gradient update required.

People behind / around the work

BPP → @austinapatel, Shuran Song @SongShuran

Would be especially interesting to hear how these teams see the tradeoff between autoregressive action modeling, diffusion policies, and behavior diversity for physical in-context learning.