Deliberative alignment: reasoning enables safer language models

Free

Teaches reasoning LLMs to explicitly reason over safety specifications

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Company

OpenAI

Visit Website GitHub (40 stars)

About Deliberative alignment: reasoning enables safer language models

Deliberative alignment is a training paradigm introduced by OpenAI to align reasoning language models (such as the o-series) with safety specifications. The approach directly teaches models the text of human-written, interpretable safety policies and trains them to reason explicitly about these specifications before generating answers. By leveraging chain-of-thought reasoning, the model reflects on user prompts, identifies relevant safety rules, and drafts responses that adhere precisely to OpenAI's policies. Notably, this method does not require human-labeled chain-of-thought data or answers. In evaluations, o1 models aligned via deliberative alignment dramatically outperform GPT‑4o on safety benchmarks, demonstrating that advances in reasoning capabilities can be harnessed to improve safety.

Key Features

Directly teaches reasoning LLMs the text of human-written safety specifications

Trains models to reason explicitly about specifications before answering

Uses chain-of-thought reasoning to reflect on user prompts

Identifies relevant text from OpenAI's internal policies

Drafts safer responses without requiring human-labeled CoTs or answers

Achieves highly precise adherence to safety policies

Outperforms GPT‑4o across a range of safety benchmarks

Pros & Cons

Pros

Does not require human-labeled chain-of-thought data or answers
Leverages reasoning capabilities to improve safety
Demonstrates state-of-the-art performance on safety benchmarks
Uses interpretable, human-written safety specifications

Cons

Requires models with strong reasoning capabilities (e.g., o-series)
May involve additional computational overhead during training and inference

Best For

Safety alignment of large language modelsJailbreak prevention through reasoning over safety policiesEnforcing content policies in AI assistantsGenerating policy-compliant responses in sensitive domains

FAQ

What is deliberative alignment?

Deliberative alignment is a training paradigm that teaches reasoning LLMs the text of human-written safety specifications and trains them to reason explicitly about these specifications before answering user prompts.

Which models use deliberative alignment?

OpenAI's o-series models (e.g., o1) are aligned using deliberative alignment.

How does deliberative alignment differ from previous alignment methods?

Unlike previous methods, deliberative alignment does not require human-labeled chain-of-thought data. Instead, it trains models to reason over interpretable safety specifications directly.

Does deliberative alignment improve safety performance?

Yes, o1 models aligned with this method dramatically outperform GPT‑4o on internal and external safety benchmarks, saturating performance on many datasets.