CriticGPT

Analysis

Why This Paper Matters

Reinforcement Learning from Human Feedback (RLHF) has been a cornerstone for aligning large language models, but it has a fundamental bottleneck: humans themselves are limited in their ability to evaluate model outputs accurately. As models become more capable, their errors grow subtler, making it harder for human raters to spot mistakes. This paper tackles that bottleneck head-on by training a dedicated "critic" model—CriticGPT—that helps humans catch errors in ChatGPT's code. The work is significant because it demonstrates a scalable way to improve human oversight, which is critical for deploying AI systems in high-stakes environments like software development.

How It Works

CriticGPT is built on GPT-4 and trained with RLHF to accept a (question, answer) pair as input and output a structured critique. The critique highlights potential problems by attaching comments to specific quotes from the answer. The training process follows a multi-step pipeline:

Step 0: Tampering – Contractors are asked to introduce subtle bugs into model-written code by editing it. They record explanations for each bug as if caught in code review. Crucially, bugs are collected adversarially: contractors verify that an LLM critic misses each bug in at least one out of three samples. This ensures the bugs are genuinely hard to find, creating a high-quality training signal.

Steps 1-4: RLHF Pipeline – For each (question, answer) pair, multiple critiques are sampled. Contractors rate them on comprehensiveness, critique-bug inclusion (CBI), hallucinated bugs or nitpicks, and overall helpfulness. A reward model is trained to predict human rankings, and the policy is optimized using PPO.

Force Sampling Beam Search (FSBS) – To navigate the tradeoff between comprehensiveness and hallucinations, FSBS generates critiques by forcing the model to produce highlighted sections with constrained sampling. The best critiques are selected based on a score combining the reward model output and a length modifier: rm_score + LENGTH_MODIFIER * num_highlights. Four length modifiers are explored, corresponding to percentiles of critique lengths. CriticGPT uses RL+FSBS at the 50th percentile, producing about four highlights on average.

Results

LLM critiques are often preferred over human-written ones and catch more inserted bugs. The tamper+RLHF pipeline dramatically improves bug detection rates, with both prompted ChatGPT and CriticGPT outperforming human annotators.

CriticGPT's RL training improves performance across model scales. Simply using a larger model also boosts bug detection, but the RL training provides additional gains.

Humans write more comprehensive critiques when assisted by LLM critics. However, human-machine teams do not increase comprehensiveness further, but they do reduce hallucination rates.

Human critiques contain far fewer nitpicks and hallucinations than LLM critiques. CriticGPT substantially reduces these rates compared to the ChatGPT baseline. Human-machine teams hallucinate and nitpick less than either CriticGPT or ChatGPT alone.

FSBS effectively navigates the tradeoff between comprehensiveness and hallucinations, allowing practitioners to tune critique behavior.

Significance

CriticGPT represents a practical step toward scalable oversight in AI systems. By training a model to critique its own outputs, the approach addresses a key limitation of RLHF—human evaluator capacity. The tampering technique ensures that training data includes genuinely challenging bugs, making the critic robust. The results show that human-machine collaboration can reduce hallucinations and nitpicks, suggesting that the best path forward may involve combining human judgment with AI assistance. This work has implications beyond code review: similar critic models could be applied to any domain where accurate evaluation of AI outputs is critical, such as medical diagnosis, legal analysis, or content moderation. The paper also highlights the importance of careful dataset construction and inference-time strategies like FSBS to balance competing quality metrics.