GeneralistJEPA

NeurIPS 2026

GeneralistJEPA

Generalizing Action-Conditioned Latent World Models with Video Model Rewards

Learn motion from video models.
Predict the future in latent space.

Haichao Zhang, Yijiang Li, Shwai He,
Van Thanh Le, Nuno Vasconcelos, Yun Fu

Paper Code BibTeX

Code release forthcoming.

Explore the methodView reported results
Vary the background.
Transfer the motion.
Same action controlFrozen video generator
Three schematic video clips with different room backgrounds and the same teal action-control path.

Shared action control Varied backgrounds

Frozen JEPA encoderTemporal Δ
Motion rewardTemporal-Δ matching
Real scene + actionLatent predictor
Video generation teaches motion offline.
Inference predicts directly in latent space.Schematic clips, not experimental video outputs.

Hold the action control fixed. Generate different backgrounds. Use temporal changes as motion rewards for latent prediction.

Broad visual knowledge.
Limited action experience.

Large visual encoders learn from diverse videos. Their action-conditioned predictors often learn from much narrower experience: fewer tasks, objects, and motions.

GeneralistJEPA explores how motion priors from a frozen, controllable video model can broaden action-conditioned latent prediction.

Visual pretraining

V-JEPA 2

Snow-covered mountain landscape Dog outdoors Yellow school bus Tennis player on court Coffee makers on a counter Busy city street and public square Musician performing on stage People preparing food in a kitchen Spider on its web Desktop workspace with a monitor, keyboard, and speakers Wooden furniture and drawers Green vegetables

>1Mhours of internet video

Broad visual representations

Frozen
encoder

Action post-training

V-JEPA 2-AC

Robot arm manipulating objects on a tabletop Robot gripper over a wooden tabletop Robot arm interacting with objects in a laboratory

<62hours of robot video

Narrower action-conditioned experience

Training scales and illustrative scenes from the accompanying presentation. The comparison motivates broader motion supervision for the latent predictor.

Can video-model motion priors broaden the predictor’s experience?

Teach motion. Keep latent prediction.

Generate varied backgrounds under shared action controls. Transfer temporal changes as motion rewards.

GeneralistJEPA training: content queries and action-dynamics keys and values fuse into a complete future prediction. Real future latents supervise that prediction, while temporal differences of the fused prediction match motion targets from a frozen, offline video teacher.
Both losses supervise the complete prediction after fusion. Real future latents anchor scene content; the frozen video teacher provides patch-level temporal changes as motion rewards. The teacher operates offline, and gradients update the latent predictor. Feature grids are schematic.

Generalization, measured.

Reported evaluations across three complementary settings.

Reported results

Results on 111 categories, including 102 unseen categories

EgoDex official test set. No test-time adaptation.

111Evaluated categories

102Unseen categories included

2,525 / 3,243Unseen / total test episodes

Macro ADE ↓ Lower is better

Bars start at zero. Macro averages over 111 categories, including 102 unseen categories.

View all EgoDex metrics
Official-test macro results on 111 categories, including 102 unseen categories. Lower ADE/FDE and higher PCK are better.
MethodADE ↓FDE ↓PCK ↑
V-JEPA 2 AC0.090110.0903629.52%
DINO-WM0.112140.1152215.34%
DINOv20.116270.1196114.61%
V-JEPA 20.091540.0923627.00%
GeneralistJEPA0.089950.0896429.52%

Read the comparison carefully. GeneralistJEPA reports the lowest ADE and FDE in this table. PCK is effectively tied with V-JEPA 2 AC. Metrics are macro-averaged over all 111 categories, including 102 unseen categories.

Source: accompanying research presentation, “Reported Results on Three Datasets.” Values are reported measurements, not an independent reproduction.

Research resources

A closer look at learning generalizable dynamics in representation space.

Paper
Read paper (PDF)
arXiv
Preprint link forthcoming
Research code
Project repository
Code release forthcoming
Qualitative videos
To be added with the release
Citation
Download BibTeX

Cite this work

GeneralistJEPA at NeurIPS 2026.

@inproceedings{zhang2026generalistJEPA,
  title = {Generalizing Action-Conditioned Latent World Models with Video Model Rewards},
  author = {Zhang, Haichao and Li, Yijiang and He, Shwai and Le, Van Thanh and Vasconcelos, Nuno and Fu, Yun},
  journal={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=E6rncT8Glg}
}