OmniFusion

Model Overview

OmniFusion is an advanced multimodal AI model designed to extend the capabilities of traditional language processing systems. It integrates additional data modalities such as images, and potentially audio, 3D and video content. The model was developed by the AIRI-Institute and is based on the open source Mistral-7B core.

OmniFusion has two versions - the first uses one visual encoder CLIP-ViT-L, while the second uses two encoders (CLIP-ViT-L and Dino V2). The key component is an adapter mechanism that allows the language model to interpret and incorporate information from different modalities.

The model was trained on a diverse dataset covering tasks like image captioning, VQA, WebQA, OCRQA, and conversational QA. This training process involves two stages - first pretraining the adapter on image captioning, then unfreezing the Mistral model for improved understanding of dialog formats and complex queries.

Model Inputs and Outputs

Inputs

• Text prompts
• Images
• Potentially audio, 3D and video content in the future

Outputs

• Multimodal responses that synthesize information from various input modalities
• Enhanced language understanding and generation capabilities compared to traditional text-only models

Capabilities

OmniFusion extends the capabilities of language models by enabling them to understand and generate responses that integrate information from multiple modalities. For example, the model can answer questions about the contents of an image, generate image captions, or engage in multimodal dialog that references both text and visual elements.

What Can I Use It For?

OmniFusion opens up new possibilities for multimodal applications, such as:

• Intelligent image-based assistants that can answer questions and describe the contents of images
• Multimodal chatbots that can engage in dialog referencing both text and visual information
• Automated image captioning and description generation
• Multimodal question answering systems that can reason about both text and visual input

Things to Try

Some interesting things to explore with OmniFusion include:

• Providing the model with a diverse set of multimodal prompts (e.g. an image plus a text question) and observing how it integrates the information to generate a response
• Evaluating the model's performance on specialized multimodal benchmarks or datasets to better understand its strengths and limitations
• Experimenting with different ways of structuring the input (e.g. using custom tokens to mark visual data) to see how it impacts the model's multimodal reasoning capabilities
• Investigating how OmniFusion compares to other multimodal models in terms of performance, flexibility, and ease of use for specific applications