A Single Backbone for Video Generation and Robot Control

Method

Each action timestep becomes one patch token in a video model’s latent space, so the transformer treats actions as video patches. The only new weights are 1,280 parameters, and one forward pass returns 16 actions with no video generated.

We co-train on three objectives. Action-only, from the current frame, is the one used at test time.

Single-step denoising

Videos have high-frequency detail like moving objects and textures. Actions are smoother. A video model learns to add detail in its last, low-noise steps, and on actions it shows as jitter. Single-step diffusion gets around this jitter.

Inference

Real robot

LIBERO held-out tasks

Citation

@inproceedings{collins2026tapestry,
  title     = {TAPESTRY: A Single Backbone for Video Generation and Robot Control},
  author    = {Collins, Jeremy A. and Lee, Seungjae and Wachakorn, Rachanon and
               Srinivasan, Krishnan and Garg, Animesh and
               Guizilini, Vitor Campagnolo and Shah, Paarth},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}