SALMON: Self-Alignment with Instructable Reward Models

Read original: arXiv:2310.05910 - Published 4/11/2024 by Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

⚙️

Overview

This paper presents a novel approach called SALMON to align base language models with minimal human supervision.
SALMON uses a small set of human-defined principles to train an instructable reward model, which can then be used to guide the reinforcement learning (RL) training of policy models.
This approach reduces the reliance on collecting high-quality human annotations and in-distribution response preferences, which are often challenging to obtain for complex tasks.
The authors apply SALMON to the LLaMA-2-70b base language model and develop an AI assistant named Dromedary-2, which significantly outperforms several state-of-the-art AI systems on various benchmark datasets.

Plain English Explanation

The paper introduces a new way to train AI language models to behave in alignment with human values and preferences. Traditionally, this has been done by having humans provide a lot of detailed feedback and examples of desired responses (Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF)). However, this can be challenging, as it's hard to get consistent and comprehensive feedback from humans, especially for complex tasks.

The key innovation in this paper is the SALMON approach, which uses a small set of high-level human-defined principles to train a "reward model." This reward model can then be used to guide the reinforcement learning (RL) process, teaching the AI assistant to behave according to those principles. This reduces the need for extensive human feedback, making the alignment process more efficient and scalable.

The authors apply SALMON to the LLaMA-2-70b language model, creating an AI assistant called Dromedary-2. With just 6 examples for context and 31 human-defined principles, Dromedary-2 outperforms several other state-of-the-art AI systems on various benchmarks. This demonstrates the power of the SALMON approach in aligning AI agents with human values and preferences using minimal supervision.

Technical Explanation

The key elements of the SALMON approach are:

Instructable Reward Model: The authors train a reward model on synthetic preference data, which can generate reward scores based on arbitrary human-defined principles. This allows them to have full control over the preferences that will guide the RL training.
RL Training with Instructable Reward: During the RL training phase, the authors can adjust the human-defined principles, and the instructable reward model will generate the corresponding rewards to shape the behavior of the policy models.
Application to LLaMA-2-70b: The authors apply the SALMON approach to the LLaMA-2-70b base language model, creating an AI assistant called Dromedary-2. They use only 6 exemplars for in-context learning and 31 human-defined principles to train Dromedary-2.
Benchmark Evaluation: Dromedary-2 is evaluated on various benchmark datasets and is shown to significantly outperform several state-of-the-art AI systems, including LLaMA-2-Chat-70b.

The key insight behind SALMON is that by using an instructable reward model, the authors can reduce the reliance on collecting high-quality human annotations and in-distribution response preferences, which are often challenging to obtain, especially for complex tasks. This allows for more efficient and scalable alignment of language models with human values and preferences.

Critical Analysis

The SALMON approach presented in this paper is a promising step towards more efficient and controllable alignment of large language models with human values. However, there are a few potential limitations and areas for further research:

Generalization of Principles: The authors demonstrate the effectiveness of SALMON using a specific set of 31 human-defined principles. It would be interesting to see how well the approach generalizes to a broader or more diverse set of principles, and how the model's performance and behavior might be affected.
Robustness to Principle Shifts: The paper does not extensively explore the model's behavior when the human-defined principles are shifted or modified during the RL training phase. It would be valuable to understand the model's sensitivity to such changes and how it might impact the resulting behavior.
Interpretability and Transparency: While the instructable reward model provides a way to control the model's preferences, the paper does not delve into the interpretability and transparency of the underlying principles and their influence on the model's decision-making process. Exploring these aspects could be important for building trust and understanding in these AI systems.
Scalability and Generalization: The authors demonstrate the effectiveness of SALMON on a specific language model and task. It would be interesting to see how well the approach scales to larger models and more diverse domains, as well as how it compares to other state-of-the-art alignment approaches, such as CodeCLM, FGAIf, and SambaLingo.

Overall, the SALMON approach presents a promising direction for aligning large language models with human values and preferences using minimal supervision. Further research and exploration of the approach's limitations and potential extensions could lead to valuable insights for the field of AI alignment.

Conclusion

The paper introduces a novel approach called SALMON that can align base language models with human values and preferences using a small set of human-defined principles. This approach reduces the reliance on high-quality human annotations and in-distribution response preferences, which are often challenging to obtain, especially for complex tasks.

By applying SALMON to the LLaMA-2-70b base language model, the authors develop an AI assistant named Dromedary-2 that significantly outperforms several state-of-the-art AI systems on various benchmark datasets. This demonstrates the potential of the SALMON approach in creating more efficient and controllable alignment of language models with human values.

While the paper presents a promising solution, there are still areas for further research, such as exploring the generalization of the approach to a broader set of principles, understanding its robustness to principle shifts, and investigating the interpretability and transparency of the underlying decision-making process. Addressing these aspects could lead to valuable insights for the field of AI alignment and the development of more trustworthy and reliable AI systems.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

⚙️

SALMON: Self-Alignment with Instructable Reward Models

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision, using only a small set of human-defined principles, yet achieving superior performance. Central to our approach is an instructable reward model. Trained on synthetic preference data, this model can generate reward scores based on arbitrary human-defined principles. By merely adjusting these principles during the RL training phase, we gain full control over the preferences with the instructable reward model, subsequently influencing the behavior of the RL-trained policy models, and reducing the reliance on the collection of online human preferences. Applying our method to the LLaMA-2-70b base language model, we developed an AI assistant named Dromedary-2. With only 6 exemplars for in-context learning and 31 human-defined principles, Dromedary-2 significantly surpasses the performance of several state-of-the-art AI systems, including LLaMA-2-Chat-70b, on various benchmark datasets. We have open-sourced the code and model weights to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, improved controllability, and scalable oversight.

4/11/2024

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, Mingyi Hong

Aligning human preference and value is an important requirement for contemporary foundation models. State-of-the-art techniques such as Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine-tuning (SFT), where the model is fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn a reward model, which is in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as a proxy to human preference, and it is critical to guide the RL step towards improving the model quality. In this work, we argue that the SFT stage significantly benefits from learning a reward model as well. Instead of using the human demonstration data directly via supervised learning, we propose to leverage an Inverse Reinforcement Learning (IRL) technique to (explicitly or implicitly) build an reward model, while learning the policy model. This approach leads to new SFT algorithms that are not only efficient to implement, but also promote the ability to distinguish between the preferred and non-preferred continuations. Moreover, we identify a connection between the proposed IRL based approach, and certain self-play approach proposed recently, and showed that self-play is a special case of modeling a reward-learning agent. Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align 1B and 7B models using proposed methods and evaluate them on a reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that it is beneficial to explicitly or implicitly leverage reward learning throughout the entire alignment process.

5/30/2024

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, Zhaoran Wang

Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named textit{Self-Exploring Language Models} (SELM), eliminates the need for a separate RM and iteratively updates the LLM with a straightforward objective. Compared to textit{Direct Preference Optimization} (DPO), the SELM objective reduces indiscriminate favor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard academic benchmarks in different settings. Our code and models are available at https://github.com/shenao-zhang/SELM.

10/11/2024

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Amrit Bedi, Furong Huang

Reinforcement Learning from Human Feedback (RLHF) is a key method for aligning large language models (LLMs) with human preferences. However, current offline alignment approaches like DPO, IPO, and SLiC rely heavily on fixed preference datasets, which can lead to sub-optimal performance. On the other hand, recent literature has focused on designing online RLHF methods but still lacks a unified conceptual formulation and suffers from distribution shift issues. To address this, we establish that online LLM alignment is underpinned by bilevel optimization. By reducing this formulation to an efficient single-level first-order method (using the reward-policy equivalence), our approach generates new samples and iteratively refines model alignment by exploring responses and regulating preference labels. In doing so, we permit alignment methods to operate in an online and self-improving manner, as well as generalize prior online RLHF methods as special cases. Compared to state-of-the-art iterative RLHF methods, our approach significantly improves alignment performance on open-sourced datasets with minimal computational overhead.

6/26/2024