Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Published 11/4/2024 by Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, Ngai Wong

Overview

• This paper examines the relationship between language model size and vocabulary size, finding that larger models perform better with larger vocabularies.

• The researchers conduct experiments on various language models, including those discussed in other papers, to understand how vocabulary size impacts model performance.

• The key insight is that as language models grow larger, they can effectively utilize larger vocabularies, which allows them to better capture the nuances and complexities of natural language.

Optimal vocabulary size scales sublinearly with non-vocabulary parameters.

1/4

Original caption: Figure 1: The relationship between non-vocabulary parameters Nnvsubscript𝑁nvN_{\rm nv}italic_N start_POSTSUBSCRIPT roman_nv end_POSTSUBSCRIPT and the corresponding optimal vocabulary parameters Nvoptsuperscriptsubscript𝑁voptN_{\rm v}^{\rm opt}italic_N start_POSTSUBSCRIPT roman_v end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_opt end_POSTSUPERSCRIPT follows a power law, where Nvoptsuperscriptsubscript𝑁voptN_{\rm v}^{\rm opt}italic_N start_POSTSUBSCRIPT roman_v end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_opt end_POSTSUPERSCRIPT should be scaled slower than Nnvsubscript𝑁nvN_{\rm nv}italic_N start_POSTSUBSCRIPT roman_nv end_POSTSUBSCRIPT as γ<1𝛾1\gamma<1italic_γ < 1. Empirical results align with predictions of our proposed approaches, with larger circles indicating higher loss values. Here V𝑉Vitalic_V refers to the vocabulary size i.e. the number of distinct tokens.

LLMs have suboptimal vocabularies, some close to optimal.

Original caption: Figure 2: Vocabulary parameters of popular LLMs and predicted optimal vocabulary parameters at a compute-optimal number of training tokens. Most current LLMs have suboptimal vocabulary parameters due to vocabulary sizes, which are smaller than the predicted optimal values. Among the current models, StarCoder2-3B, OLMo-7B, InternLM2-20B, and Gemma2-27B have vocabulary sizes that come closest to the optimal allocation for their respective model sizes.

Optimal vocabulary size minimizes FLOPs and loss, increasing with budget.

Original caption: Figure 3: Left: FLOPs curve with various vocabulary sizes, assuming all configurations achieve a fixed loss. There exists an optimal vocabulary size that minimizes FLOPs. Right: Loss curves with various vocabulary sizes given different FLOP budgets. For each budget there exists an optimal vocabulary size that minimizes loss. As the FLOP budget increases this optimal vocabulary size increases (shifts to the right).

Optimal vocabulary parameters and size, by three approaches, given non-vocabulary parameters.

1/2

Model Size (Parameters)	Number of Operations (OPT 1)	Number of Operations (OPT 2)	Number of Operations (OPT 3)	Dimension	Memory (OPT 1)	Memory (OPT 2)	Memory (OPT 3)	FLOPs Budget
3B	0.1B	0.1B	0.1B	3200	39K	43K	37K	1.3e+21
7B	0.3B	0.3B	0.2B	4096	62K	67K	60K	7.1e+21
13B	0.4B	0.5B	0.4B	5120	83K	91K	81K	2.4e+22
30B	0.9B	0.9B	0.9B	6048	142K	154K	142K	1.3e+23
70B	1.7B	1.9B	1.8B	8192	212K	231K	218K	7.1e+23
130B	2.9B	3.2B	3.0B	12888	237K	258K	248K	2.4e+24
300B	5.8B	6.4B	6.3B	16384	356K	389K	383K	1.3e+25

Original caption: Table 1: We report the predicted optimal vocabulary parameters Nvsubscript𝑁𝑣N_{v}italic_N start_POSTSUBSCRIPT italic_v end_POSTSUBSCRIPT and the vocabulary size V𝑉Vitalic_V by the proposed three approaches given Nn⁢vsubscript𝑁𝑛𝑣N_{nv}italic_N start_POSTSUBSCRIPT italic_n italic_v end_POSTSUBSCRIPT. We assume the training FLOPs are optimally allocated i.e. that the non-vocabulary parameters and training data are scaled equally. “App” denotes the approach.

V	Nv	D	H	ARC-C	ARC-E	Hellaswag	OBQA	WG	PIQA	BoolQ	Average
FLOPs Budget 1.2e21 (Optimally-Allocated Training Data)
32K	0.10B	67.3B	266.6B	28.5 ± 1.3	49.2 ± 1.0	47.5 ± 0.5	31.6 ± 2.1	50.4 ± 1.4	71.4 ± 1.1	56.4 ± 0.9	47.9
35K	0.11B	67.1B	268.2B	29.1 ± 1.3	50.6 ± 1.0	48.1 ± 0.5	31.6 ± 2.1	51.9 ± 1.4	71.4 ± 1.1	57.1 ± 0.9	48.5

Original caption: Table 2: Zero-shot performance of models with Nn⁢v=2.87⁢Bsubscript𝑁𝑛𝑣2.87𝐵N_{nv}=2.87Bitalic_N start_POSTSUBSCRIPT italic_n italic_v end_POSTSUBSCRIPT = 2.87 italic_B comparing the commonly used V=32⁢K𝑉32𝐾V=32Kitalic_V = 32 italic_K with our predicted optimal vocabulary Vo⁢p⁢tsuperscript𝑉𝑜𝑝𝑡V^{opt}italic_V start_POSTSUPERSCRIPT italic_o italic_p italic_t end_POSTSUPERSCRIPT. We consider the scenario where the number of training data is equally scaled with the non-vocabulary parameters. We report accuracy and standard deviation in percentages. Accuracy is normalized: The predicted likelihoods are divided by the length of each choice for multiple choices to eliminate the effect of text length on predictions.

Plain English Explanation

The paper investigates how the size of a language model, which is a type of artificial intelligence that can understand and generate human-like text, affects the optimal size of its vocabulary. The researchers found that as language models become larger and more capable, they perform better when they have access to a larger vocabulary.

This is because larger models have the capacity to effectively learn and utilize a richer set of words and expressions. With a larger vocabulary, they can more accurately represent the subtleties and variations in natural language.

For example, a small language model may only know a few basic ways to express a concept, like "happy" or "joyful." But a larger model with a more extensive vocabulary could choose from a wider range of nuanced words like "elated," "ecstatic," "gleeful," and so on, allowing it to generate more natural and human-like text.

The researchers provide evidence for this relationship between model size and vocabulary size through a series of experiments. They show that as language models grow larger, the optimal vocabulary size also increases, allowing the models to achieve better performance on various language tasks.

Technical Explanation

The paper investigates the relationship between the size of language models and the size of their vocabularies. The researchers conduct experiments using a variety of large language models, including those discussed in related papers like Language Models Scale Reliably with Training Data Size, to understand how vocabulary size impacts model performance.

The key finding is that as language models become larger, they are able to effectively utilize larger vocabularies, which allows them to better capture the nuances and complexities of natural language. This is because larger models have the capacity to learn and leverage a richer set of words and expressions, enabling them to more accurately represent the subtle variations in human language.

The researchers systematically explore this relationship by training language models of different sizes and measuring their performance on various tasks as a function of vocabulary size. They find that the optimal vocabulary size increases as the model size grows, and that larger models consistently outperform smaller models when given access to a vocabulary that is appropriately scaled to their size.

These results have important implications for the design and development of large language models. They suggest that as these models continue to grow in size and capability, it will be necessary to also scale up their vocabularies to unlock their full potential and achieve the best possible performance on natural language tasks.

Critical Analysis

The paper provides a compelling and well-designed study on the relationship between language model size and vocabulary size. The researchers' approach of systematically exploring this relationship across multiple model architectures and tasks is a strength, as it strengthens the generalizability of their findings.

However, one potential limitation is that the experiments were conducted on a relatively narrow set of language tasks, such as language modeling and machine translation. It would be interesting to see how the insights from this paper translate to other areas of natural language processing, such as question answering, dialogue systems, or text generation for creative applications.

Additionally, the paper does not delve deeply into the underlying mechanisms that drive the observed relationship between model size and vocabulary size. Further research could investigate the cognitive and computational processes that enable larger models to effectively leverage larger vocabularies, which could lead to a more fundamental understanding of language model scaling.

Another area for potential exploration is the interplay between vocabulary size and other model hyperparameters, such as the number of model parameters or the training dataset size. It's possible that there are complex interactions between these factors that could provide additional insights into the design of large language models.

Overall, this paper represents an important contribution to the growing body of research on scaling laws in language models. By highlighting the significance of vocabulary size as a key factor in model performance, it encourages the AI research community to consider vocabulary as a critical component in the development of ever-larger and more capable language models.

Conclusion

This paper presents compelling evidence that as language models become larger and more sophisticated, they are able to effectively utilize larger vocabularies, which in turn allows them to better capture the nuances and complexities of natural language.

The researchers' systematic exploration of this relationship across multiple model architectures and tasks provides a strong foundation for understanding the importance of vocabulary size in the development of large language models. Their findings suggest that as these models continue to grow in size and capability, it will be necessary to also scale up their vocabularies to unlock their full potential and achieve the best possible performance on a wide range of natural language tasks.

While the paper focuses on a relatively narrow set of language tasks, the insights it provides have broader implications for the field of natural language processing. By highlighting the significance of vocabulary size as a key factor in model performance, it encourages the AI research community to consider vocabulary as a critical component in the design and development of ever-larger and more capable language models.

Full paper

Loading PDF viewer...

Read original: arXiv:2407.13623

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →