The Missing Variable in Robot Benchmarking: The Task Itself

Robot-learning benchmarks are becoming larger, more realistic, and more diverse.

They randomize object poses.
They change textures and lighting.
They introduce unseen layouts.
They hold out objects.
They evaluate compositional behaviors.
Some even procedurally generate entire scenes.

But there is still a deeper problem:

in most benchmarks, the semantic task itself is already known before evaluation begins.

That distinction matters much more than it first appears.

1. Randomizing the world is not the same as randomizing the task

A useful way to separate the two is:

τ = (E, O, N, G)

where:

• E = environment / structural scene
• O = object types and semantic roles
• N = object counts
• G = goal program / composition of subgoals

An episode additionally has an initial physical state:

s₀ ~ p(s₀ | τ)

This gives two fundamentally different kinds of generalization.

State-space generalization

Keep the task fixed:

τ = constant

but sample another initial state:

s₀' ~ p(s₀ | τ)

Examples:

• move the apple 10 cm
• rotate the cup
• change lighting
• change camera pose
• add pose jitter
• change background texture

These are important robustness tests.

But they are still the same task.

Task-space generalization

Change the task specification itself:

τ_eval ≠ τ_train

Examples:

pick up toy

→

pick up apple

or

pick apple

→

open drawer -> pick apple -> place apple inside -> close drawer

The second example is especially important.

A change in natural-language wording alone is not necessarily a new task.

What matters is whether the required goal structure or subgoal composition changes.

2. What should count as a "new task"?

A useful criterion is that the complete task tuple

(E, O, N, G)

must represent a combination absent from training.

Novelty can come from one or more axes:

This is different from simply changing continuous state variables.

apple at x=0.3

and

apple at x=0.5

should not be treated as two different tasks.

3. The weaknesses of current robot benchmarks

Several recurring limitations appear across modern manipulation benchmarks.

Fixed semantic task universes

Many benchmarks define a finite set of task templates first and then divide them into train/test splits.

The test tasks may be unseen during training, but the entire ontology is still manually designed beforehand.

This tests generalization inside a closed task universe.

"Unseen" often means unseen combination, not newly generated task

A benchmark may hold out:

object B + goal C

while training on:

object A + goal C

and

object B + goal D.

That is useful compositional generalization.

But the evaluator is still selecting from a predefined combinatorial space.

Procedural generation often stops at the scene level

Modern simulators can generate enormous numbers of episodes:

• different layouts
• object poses
• textures
• distractors
• articulation states
• lighting
• cameras

Yet the underlying task can remain:

put object X into container Y

for every single episode.

An infinite state distribution does not automatically create an infinite task distribution.

Static test sets eventually become part of the training ecosystem

Once a benchmark is public, its:

• task definitions
• object library
• instructions
• success predicates
• simulator implementation
• evaluation distribution

can influence model development.

Even without intentionally training on the test set, benchmark-specific optimization becomes increasingly possible.

This is one reason static leaderboards eventually saturate.

4. Why demonstration-conditioned learning makes this issue even more important

In in-context imitation learning, behavior prompting, or demonstration-conditioned policies, the demonstration is supposed to specify:

"What should the robot do?"

A typical evaluation looks like:

demonstration + current observation -> robot action

But there are two very different ways to construct that test.

Closed task evaluation

The benchmark has a fixed collection of known task families.

A demonstration selects one of them.

The scene changes, but the semantic task universe remains fixed.

Open task evaluation

The evaluator first generates a new semantic task.

Then it generates or selects an appropriate demonstration for that task.

The policy must infer and execute a task whose exact combination did not exist in training.

The latter is much closer to the strongest interpretation of in-context robot learning.

5. How current benchmarks compare

The important question is therefore not simply:

"Does this benchmark randomize?"

The better question is:

"At which level does it randomize?"

✓ = strong support
△ = partial / constrained support

The pattern is surprisingly consistent.

The field has become very good at generating:

s₀

but much less standardized at generating:

τ

6. The interesting research direction is therefore not "more domain randomization"

The missing benchmark primitive is closer to a task compiler.

Instead of defining all evaluation tasks before training, evaluation could work like this:

task generator
    ↓
semantic validator
    ↓
physical-feasibility validator
    ↓
success-predicate compiler
    ↓
scene generator
    ↓
demonstration generator / selector
    ↓
closed-loop evaluation

The task itself could be produced only after model commitment.

For example:

Training tasks

• put apple on plate
• put toy in box
• open drawer
• move cup to shelf

Generated evaluation task

• open drawer
• place apple inside
• place toy beside the apple
• close drawer

The primitives are familiar.

The exact composition, object-role assignment, environment, count, and goal graph are not.

This creates a much stronger test of whether a model learned reusable behavioral structure rather than benchmark-specific mappings.

7. The difficult part is not generating the task text

LLMs can already produce things like:

Pick up the apple and place it inside the second drawer.
Then move the cup onto the plate.
Finally close the drawer.

Generating equivalent BDDL, Python, JSON, YAML, or another simulator-specific task representation is also increasingly practical.

The hard problems are downstream:

1. Is the generated task physically possible?
2. Are all referenced objects actually available?
3. Is there a valid initial configuration?
4. Does the subgoal ordering make sense?
5. Can success be checked objectively?
6. Is the task truly novel relative to training?
7. Is difficulty controlled?
8. Can independent evaluators reproduce it?

Recent systems such as RoboWits, GenSim, RoboLab, AXIS, RoboPlayground, PhyRoGen, and related task-generation frameworks already provide many of these components individually.

The remaining opportunity is to connect them directly to the evaluation protocol.

8. A stronger benchmark definition

Instead of publishing a test set:

D_test = {τ₁, τ₂, ..., τₙ}

publish a task distribution and generator:

τ_eval ~ p_generator(τ)

subject to constraints such as:

τ_eval ∉ D_train

and

Valid(τ_eval) = 1

Evaluation becomes:

model commit
    ↓
future random seed
    ↓
generate hidden task specification
    ↓
validate novelty
    ↓
validate solvability
    ↓
instantiate environment
    ↓
generate/select demonstration
    ↓
run closed-loop policy
    ↓
evaluate simulator-state success predicates

This changes the meaning of a benchmark considerably.

The evaluator no longer asks:

"Can this model solve the held-out tasks that were designed months ago?"

It asks:

"Can this model solve a valid task that was created after the model was submitted?"

9. Conclusion

The next step in robot benchmarking may not simply be larger datasets, more objects, more cameras, or more aggressive domain randomization.

Those primarily expand the state distribution.

The more fundamental challenge is expanding the task distribution itself.

A strong future benchmark should be able to vary:

E, O, N, G

and particularly the structure of:

G = subgoal program

The key distinction is simple:

New pose != new task.

New texture != new task.

New camera != new task.

Even a new object is only one axis of novelty.

A genuinely strong evaluation asks whether a robot can execute a new composition of goals, objects, roles, and environments that did not exist before evaluation started.

Once task generation, semantic validation, physical feasibility checking, success-predicate construction, and procedural scene generation are connected into one evaluator, robot benchmarks can move from:

static test sets

to:

distributions over executable tasks.

That may be a much better foundation for measuring real robot generalization.