Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

2404.14219

YC

410

Reddit

0

Published 4/24/2024 by Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl and 77 others
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Abstract

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

Get summaries of the top AI research delivered straight to your inbox:

Overview

  • Introduces a highly capable language model, Phi-3, that can run locally on a cell phone
  • Provides technical details on the model's architecture, performance, and capabilities
  • Explores the potential benefits and challenges of deploying large language models on mobile devices

Plain English Explanation

This research paper describes a new language model called Phi-3 that can run directly on a cell phone, without needing to connect to the internet or a remote server. Highly capable language models like GPT-3 have shown impressive abilities at tasks like answering questions, summarizing text, and generating human-like writing. However, these models are usually very large and require significant computing power to run, making them difficult to deploy on everyday mobile devices.

The key innovation of Phi-3 is that it is designed to deliver high performance and capability while still being small enough to run locally on a cell phone. This means you could use advanced language AI features like natural language generation or question answering without needing an internet connection or cloud computing resources. The researchers achieved this by carefully optimizing the model architecture and training process to balance size, speed, and accuracy.

If successful, Phi-3 could pave the way for a new generation of highly capable AI assistants that can run directly on our smartphones and other mobile devices, without the need to send our data to the cloud. This could have important implications for privacy, security, and accessibility, especially in areas with unreliable internet access. However, there are also technical challenges in making such a powerful model run efficiently on limited hardware.

Technical Explanation

The Phi-3 model is built using a transformer-based architecture, similar to large language models like GPT-3, but with several key optimizations to reduce the model size and improve efficiency. These include:

  • Lightweight attention mechanisms: The model uses a more efficient attention module design compared to standard transformers, reducing the number of parameters required.
  • Knowledge distillation: The researchers trained Phi-3 by distilling knowledge from a larger teacher model, allowing it to achieve high performance with a much smaller model size.
  • Quantization: The model weights are quantized to lower precision data types (e.g. 8-bit integers), further reducing the memory footprint without significant accuracy loss.

Through these and other optimizations, the final Phi-3 model is able to achieve state-of-the-art performance on a range of language tasks while being small enough to run locally on a smartphone processor. The researchers report that Phi-3 has a model size under 500MB and can perform inference in under 500ms, making it viable for real-time applications.

The paper also includes detailed evaluations of Phi-3's performance, comparing it to other compact language models such as Octopus V3 and TinyGPT-V. The results demonstrate Phi-3's ability to match or exceed the accuracy of these other models while being significantly smaller in size.

Critical Analysis

The Phi-3 research represents an important step towards making highly capable language AI models practical for deployment on mobile and edge devices. By addressing the challenges of model size and computational efficiency, the researchers have shown it is possible to bring cutting-edge natural language processing capabilities directly to users' fingertips.

However, the paper does not deeply explore some potential limitations and tradeoffs of this approach. For example, it is unclear how Phi-3's performance would scale to more complex or open-ended language tasks compared to larger cloud-based models. There may also be challenges in keeping the model up-to-date and adapting it to new domains without access to the compute resources available in the cloud.

Additionally, while the focus on privacy and accessibility is commendable, the paper does not address potential misuse or societal impacts of having such powerful language AI running directly on user devices. Issues around algorithmic bias, data privacy, and the responsible development of these technologies should be carefully considered.

Overall, the Phi-3 research represents an exciting step forward, but follow-up work will be needed to fully realize the potential benefits and mitigate the risks of deploying large language models on mobile devices.

Conclusion

The Phi-3 technical report describes a highly capable language model that can run locally on a cell phone, overcoming the typical size and performance constraints of deploying such models on mobile hardware. This innovation has the potential to enable a new generation of advanced AI assistants that operate directly on user devices, without the need for an internet connection or cloud computing resources.

By carefully optimizing the model architecture and training process, the researchers have demonstrated that it is possible to achieve state-of-the-art natural language processing capabilities in a compact, efficient package. If successful, this work could have far-reaching implications for privacy, accessibility, and the real-world deployment of large language AI models.

However, the paper also highlights the need for further research to fully address the challenges and potential risks of this approach. Ongoing work will be required to ensure these technologies are developed and deployed responsibly, with a focus on user safety, algorithmic fairness, and the broader societal impact.



This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Related Papers

Octopus v2: On-device language model for super agent

Octopus v2: On-device language model for super agent

Wei Chen, Zhiyuan Li

YC

0

Reddit

0

Language models have shown effectiveness in a variety of software applications, particularly in tasks related to automatic workflow. These models possess the crucial ability to call functions, which is essential in creating AI agents. Despite the high performance of large-scale language models in cloud environments, they are often associated with concerns over privacy and cost. Current on-device models for function calling face issues with latency and accuracy. Our research presents a new method that empowers an on-device model with 2 billion parameters to surpass the performance of GPT-4 in both accuracy and latency, and decrease the context length by 95%. When compared to Llama-7B with a RAG-based function calling mechanism, our method enhances latency by 35-fold. This method reduces the latency to levels deemed suitable for deployment across a variety of edge devices in production environments, aligning with the performance requisites for real-world applications.

Read more

4/17/2024

🎯

ChuXin: 1.6B Technical Report

Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, Zhihua Wu

YC

0

Reddit

0

In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weights and architecture, we have made everything needed to train a model available, including the training data, the training process, and the evaluation code. Our goal is to empower and strengthen the open research community, fostering transparency and enabling a new wave of innovation in the field of language modeling. Furthermore, we extend the context length to 1M tokens through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. The weights for both models are available at Hugging Face to download and use.

Read more

5/9/2024

PHUDGE: Phi-3 as Scalable Judge

PHUDGE: Phi-3 as Scalable Judge

Mahesh Deshwal, Apoorva Chawla

YC

0

Reddit

0

In this paper cum technical report, we present PHUDGE A fine tuned Phi3 model that achieved SOTA results in 4 tasks as Feedback Test, Feedback OOD, MT Human, Preference Test surpassing each and every existing model in latency and throughput. It shows very strong correlation not only with GPT4 but with Human annotators too in unseen data as well as in both absolute and relative grading tasks. We have not only addressed the usage of small LMs for cost effective production grade systems but have also shown that Causal modelling is not only slow in nature but sometimes it can hinder models learning capabilities and should be replaced by simpler tasks whenever we can to make the overall system faster and better. We show that by following systematic ML experimentation, thoughtful data augmentation and re purposing the problem itself, we can even beat 10x bigger models even with lesser training data. To the best of our knowledge, we are re the first one to experiment and showcase the usage of generalised version of Earth Movers Distance AKA Wasserstein distance by using Minkowski Distance with a penalty to control loss smoothing and can be used as a loss function instead of Cross Entropy to get stable training and better results for grading tasks.

Read more

5/16/2024

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, Lichao Sun

YC

0

Reddit

0

In recent years, multimodal large language models (MLLMs) such as GPT-4V have demonstrated remarkable advancements, excelling in a variety of vision-language tasks. Despite their prowess, the closed-source nature and computational demands of such models limit their accessibility and applicability. This study introduces TinyGPT-V, a novel open-source MLLM, designed for efficient training and inference across various vision-language tasks, including image captioning (IC) and visual question answering (VQA). Leveraging a compact yet powerful architecture, TinyGPT-V integrates the Phi-2 language model with pre-trained vision encoders, utilizing a unique mapping module for visual and linguistic information fusion. With a training regimen optimized for small backbones and employing a diverse dataset amalgam, TinyGPT-V requires significantly lower computational resources 24GB for training and as little as 8GB for inference without compromising on performance. Our experiments demonstrate that TinyGPT-V, with its language model 2.8 billion parameters, achieves comparable results in VQA and image inference tasks to its larger counterparts while being uniquely suited for deployment on resource-constrained devices through innovative quantization techniques. This work not only paves the way for more accessible and efficient MLLMs but also underscores the potential of smaller, optimized models in bridging the gap between high performance and computational efficiency in real-world applications. Additionally, this paper introduces a new approach to multimodal large language models using smaller backbones. Our code and training weights are available in url{https://github.com/DLYuanGod/TinyGPT-V}.

Read more

4/8/2024