A robot powered by the GEN‑1.5 model used a banana instead of a brush
- GEN‑1.5 teaches robots from a single demonstration
- The model uses multimodal data for control
- The robot replaced a brush with a banana, gathering blocks
- Tests confirm the system's flexibility and improvisation
The US-based company Generalist AI announced the development of a multimodal foundational model called GEN‑1.5. This model is designed for controlling robots and is characterized by its ability to master new physical actions after a single short demonstration placed in a context window, or after a small fine-tuning using a gradient method in a single or a few steps.
How GEN‑1.5 Works
GEN‑1.5 combines visual, textual, and sensory data, allowing the system to 'see' and 'understand' the environment in real time. When a short video demonstration of a new action is provided in the context window, the model forms an internal representation of the task and generates control signals for the robot's actuators. If necessary, the model can be further fine-tuned using a gradient method, which requires only a few iterations and a small amount of data.
Testing and Results
During a series of laboratory tests, the robot, controlled by GEN‑1.5, demonstrated the ability to improvise. In one scenario, the robot was tasked with collecting a set of blocks and placing them in a bowl. The usual brush, designed for picking up small items, was unavailable for the robot to use. The model quickly reassessed the situation and suggested an alternative – to use a banana lying nearby as an improvised tool.
The robot successfully replaced the brush with the banana, collected all the blocks, and transferred them to the bowl, thereby demonstrating the flexibility and adaptability of behavior characteristic of more advanced forms of artificial intelligence. This experiment confirms that GEN‑1.5 can not only replicate demonstrated actions but also find creative solutions in new conditions.
Such opportunities open up prospects for application in real industrial and service tasks, where conditions often change and available tools may be limited. The robot's ability to quickly adapt without lengthy retraining reduces programming costs and increases automation efficiency.
Developers emphasize that the model remains fundamental and can be integrated into various robotic platforms. Further expansion of GEN‑1.5's capabilities is planned, including more complex manipulations and operation in conditions of limited visual information.
A detailed description of the model and test results has been published on the official Generalist AI blog, which also presents technical details of the architecture and examples of application in other scenarios.
Source: N+1
