OmniJARVIS: Unified VisionLanguageAction Enables OpenWorld Instruction

Submit
github.io

OmniJARVIS: Unified vision-language-acti... enables open-world instruction

AI summary of the linked article

OmniJARVIS is a Vision-Language-Action model that its authors present as enabling open-world instruction-following agents in Minecraft, combining strong reasoning with efficient decision-making through unified tokenization of multimodal interaction data. The model adds self-supervised behavior tokens, learned by a tokenizer modified from GROOT's VAE approach using Finite Scalar Quantization, to a pretrained multimodal language model's vocabulary. According to the article, OmniJARVIS can reason through chain-of-thought, plan, answer questions, and act, and it re-reasons every 128 steps. The article reports strong performance on atomic, programmatic, and open-ended tasks in Minecraft.

No comments yetWhat caught your eye? Start the discussion.