Neura Market
Constitutional AI logo

Constitutional AI

Free

Safer model alignment.

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Visit Website GitHub (40 stars)

About Constitutional AI

Constitutional AI is a method for training harmless AI assistants through self-improvement without requiring human labels identifying harmful outputs. The only human oversight is a list of rules or principles, referred to as a 'constitution'. The process involves two phases: a supervised learning (SL) phase where an initial model generates self-critiques and revisions, then is finetuned on the revised responses; and a reinforcement learning (RL) phase that uses AI feedback to train a preference model, enabling RL from AI Feedback (RLAIF). The resulting assistant is harmless yet non-evasive, engaging with harmful queries by explaining its objections. Both phases can leverage chain-of-thought reasoning to improve transparency and human-judged performance. The method enables more precise control of AI behavior with far fewer human labels.

Key Features

Self-improvement without human-labeled harmful outputs
Uses a constitution (list of rules or principles) as sole human oversight
Two-phase training: supervised finetuning with self-critiques and revisions, then RL from AI Feedback (RLAIF)
Leverages chain-of-thought reasoning for improved transparency and performance
Trains a harmless but non-evasive AI assistant that explains objections to harmful queries
Reduces need for human labels while enabling precise behavior control

Pros & Cons

Pros
  • Dramatically reduces reliance on human labeling for harmlessness
  • Produces a non-evasive AI that engages with harmful queries respectfully
  • Chain-of-thought reasoning enhances decision transparency
  • Allows precise control over AI behavior using a small set of principles
  • Combines supervised and reinforcement learning for robust alignment
Cons
  • Requires careful design of the constitutional principles
  • Dependent on quality of the initial model and AI feedback models
  • Two-phase training may be computationally expensive
  • Effectiveness may vary across different types of harmful content

Best For

Training harmless AI assistantsAI alignment and safety researchReducing harmful and toxic outputs in language modelsCreating AI systems that can refuse harmful requests with reasoned explanations

FAQ

What is Constitutional AI?
Constitutional AI is a method for training harmless AI assistants through self-improvement, using only a list of rules or principles (a 'constitution') as human oversight, without needing human labels for harmful outputs.
How does Constitutional AI differ from RLHF?
Constitutional AI uses AI feedback (RLAIF) instead of human feedback for the RL phase, reducing the need for human labels. It also includes a supervised finetuning phase with self-critiques and revisions based on the constitution.
What are the two phases of Constitutional AI?
The supervised learning phase involves sampling from an initial model, generating self-critiques and revisions, and finetuning on revised responses. The reinforcement learning phase uses AI feedback to train a preference model, then trains with RL using that model as the reward signal.
Does Constitutional AI produce a harmless model?
Yes, the method aims to train a harmless but non-evasive AI assistant that explains its objections to harmful queries rather than ignoring or evading them.
Can Constitutional AI use chain-of-thought reasoning?
Yes, both the supervised and reinforcement learning phases can leverage chain-of-thought style reasoning to improve human-judged performance and transparency of AI decision making.