CoachSI: Self-Improvement of Robot Policies
with a Language Coach

Jiachen Tao1 Gengyu Zhang1 Haoxuan Wang1 Bing Liu1 Yan Yan1

1University of Illinois at Chicago

1frozen VLM as the coach
0new human demonstrations
0coach calls at test time
Fill Kettle
Heat Mug
Retrieve Ice Tray
Transport Cookware
Warm Croissant
Dessert Upgrade

CoachSI on long-horizon tasks in unseen scenes, from the task instruction alone, with no coach at test time. Videos play at 6× to 12× speed.

CoachSI lets a robot policy improve itself with a language coach. A frozen VLM watches the robot and names the next step; fine-tuned on its coached successes, the policy completes long-horizon tasks on its own.

Abstract

Generalist robot policies execute atomic skills reliably but fail when the same skills are composed into long-horizon tasks. Self-improvement from the policy's own experience does not close this gap: experience collected by the policy alone stays within its own distribution and reproduces what it can already do. We show that what the policy lacks is not a skill but the next step. We present CoachSI, a self-improvement method in which a frozen, off-the-shelf vision-language model (VLM) coaches the policy in language: it watches the robot and names the next step in the format of the policy's atomic instructions, and never outputs an action. Successful coached rollouts are relabeled in hindsight with the original task instruction and used to fine-tune the policy. The fine-tuned policy is then deployed on its own. On states where a World Action Model and a Vision-Language-Action model have stalled, naming the next step raises subgoal completion by 4.6× over the unchanged task instruction, and the frozen VLM supplies it from the same camera images the policy sees. After fine-tuning, unassisted success on long-horizon tasks in unseen scenes rises from 47.4% to 59.3%, 9.8 points above fine-tuning on human demonstrations and 7.3 points above fine-tuning on self-collected successes at an equal data budget. With a coach that sees only camera images, it reaches 56.8%, above both baselines. The gain is obtained with the action backbone frozen: language coaching teaches the policy what should happen next, and the skills it already has do the rest.

Three panels: the pretrained policy stalls under the task instruction; a frozen VLM coach names the next step and the same policy completes the task; fine-tuned on its coached successes, the policy completes the task alone. Below, bar charts of the main results.

Method

Coached collection

At every action chunk, a frozen VLM coach reads the camera views and selects the policy's instruction from the next-step candidates of the task, or keeps the task instruction. The policy weights are fixed and every action is the policy's own.

Hindsight relabeling and fine-tuning

Successful rollouts keep their observations and actions, and the coach's instructions are replaced by the task instruction. Fine-tuning mixes them with human demonstrations and updates only the video backbone. The action backbone stays frozen.

Deployment

The fine-tuned policy receives its observations and the task instruction alone. No human operator, no reward model, and no VLM at test time.

Overview of CoachSI: coached collection with a frozen VLM coach and a frozen policy; relabeling and fine-tuning of the video backbone only; deployment of the fine-tuned policy with no coach.

One next-step instruction resolves the stall

A stalled policy stays stalled under the task instruction. Across 500 stalled states from a World Action Model and a Vision-Language-Action model, it completes the next subgoal within 320 steps in 2.9% of continuations. Naming the next step raises that to 13.5%, a 4.6× gain, and a frozen VLM that sees only the camera images recovers 91% of it.

Each clip replays one recorded rollout up to the stall. From that state, the left side keeps the task instruction and the right side receives the coach's next step.

Share of stalled states completing the next subgoal over steps after the stall, for the task instruction, the oracle next-step instruction and the VLM coach; and the share of the oracle gain the VLM coach recovers: 101 percent with state, 91 percent from images only.
Share of stalled states in which the policy completes the next subgoal, by steps after the stall, and the share of the oracle gain that a VLM coach recovers.

A frozen VLM coaches data collection

The coach watches the robot from the first step and tells the policy the next step. It never outputs an action. The successes it produces are ones the policy does not reach alone, and they become training data once relabeled with the task instruction.

Both sides start from the same initial state. Left: the pretrained policy alone. Right: the same policy with the image-only coach.

Coached experience becomes unassisted capability

After fine-tuning, the policy receives only the task instruction. On 18 long-horizon tasks in unseen scenes, unassisted success rises from 47.4% to 59.3%. Human demonstrations and the policy's own successes, at the same data budget, reach 49.4% and 51.9%.

Pretrained policy47.4%
+ Human demonstrations49.4%
+ Self-collected successes51.9%
CoachSI, image-only coach56.8%
CoachSI, state-based coach59.3%
Unassisted success on 18 long-horizon tasks in unseen scenes. Task instruction only, no coach at test time; 1,080 episodes per policy.

Same initial state, same task instruction, no coach. Left: the pretrained policy. Right: CoachSI.

Language coaching taught the policy the next step. The skills it already had did the rest.

BibTeX

@misc{tao2026coachsiselfimprovementrobotpolicies,
  title         = {{CoachSI}: Self-Improvement of Robot Policies with a Language Coach},
  author        = {Jiachen Tao and Gengyu Zhang and Haoxuan Wang and Bing Liu and Yan Yan},
  year          = {2026},
  eprint        = {2611.25529},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2611.25529}
}