Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

Analysis

Why This Paper Matters

Large language models (LLMs) are known to inherit and amplify social biases from training data, leading to harmful stereotypes in downstream applications. Existing debiasing methods often require fine-tuning the model or post-hoc editing, which can be costly or degrade performance. This paper addresses a critical gap by proposing a decoding-time debiasing approach that uses process reward models (PRMs) to steer generation away from biased outputs without altering the base model. This is significant because decoding-time interventions are model-agnostic and can be applied on top of any existing LLM, making them highly practical for real-world deployment.

The paper extends the scope from controlled fill-in tasks (e.g., completing a sentence with a gender-neutral pronoun) to open-ended generation (e.g., writing a story or article). This is a major step forward, as open-ended generation is more challenging due to the lack of a clear target and the need to maintain coherence over long passages. By leveraging process-level rewards, the method can intervene at intermediate steps, preventing bias from accumulating during generation.

Technical Contributions

• Process Reward Models (PRMs): The core innovation is using a reward model that evaluates the bias level of partial sequences, not just the final output. This allows for fine-grained control during decoding.
• Decoding-Time Integration: The PRM is integrated into the decoding process (e.g., beam search or sampling) to rerank or filter candidate tokens at each step, effectively guiding the generation toward less biased paths.
• Versatility: The method is demonstrated on both fill-in and open-ended generation tasks, showing its adaptability to different generation scenarios.
• Bias Mitigation: The approach targets multiple social bias dimensions, including gender and race, and is designed to reduce stereotype reinforcement.

Results

The abstract is truncated, but based on the provided text, the method shows promise in reducing social biases. The paper likely reports quantitative metrics such as stereotype scores (e.g., from BBQ or StereoSet benchmarks) and generation quality metrics (e.g., perplexity or BLEU). The key result is that the proposed method outperforms baseline debiasing techniques, achieving lower bias while maintaining fluency and coherence. However, specific numbers are not available in the abstract, so a detailed comparison is not possible here.

Significance

This work contributes to the growing field of safe and fair AI by offering a practical tool for bias mitigation in LLMs. The decoding-time approach is particularly valuable because it does not require retraining the base model, making it easy to adopt in production systems. Moreover, the use of process reward models opens new research directions for process-level control in text generation, potentially applicable to other alignment goals such as factuality or toxicity reduction. As LLMs become more pervasive, methods like this will be essential to ensure they are used responsibly and ethically.