REFINE-AF

Analysis

Why This Paper Matters

Instruction fine-tuning (IFT) has become a cornerstone for aligning large language models (LLMs) with user intent. However, creating high-quality instruction datasets typically demands extensive human annotation, which is expensive and time-consuming. REFINE-AF addresses this bottleneck by introducing a semi-automated framework that leverages reinforcement learning from automated feedback (RLAF) to generate diverse and effective instruction-input-output triplets from a minimal seed set. This matters because it offers a scalable path to produce task-agnostic instruction data, reducing human effort while improving model alignment.

How It Works

REFINE-AF operates in three stages. First, Instruction Generation starts with 175 human-written seed instructions. At each iteration, 8 instructions are sampled (6 human, 2 previously generated) as in-context examples for an LLM to generate new instructions. To ensure diversity, a new instruction is added only if its ROUGE-L similarity with any existing instruction is below 0.7, and instructions containing keywords like "image" or "graph" are filtered out.

Second, Reinforcement Learning from Automated Feedback replaces human preference judgments with an automated reward model. The reward for a triplet (instruction, input, output) is computed as:

This score combines the oasst-rm-pythia-1.4b model reward, naturalness, and coherence, while penalizing complexity (understandability). The LLM is fine-tuned using PPO to maximize this reward, aligning its generation with automated preferences.

Third, Instance Generation uses the trained LLM to produce an (input, output) pair for each instruction in the pool, yielding a complete IFT dataset. This dataset is then used for supervised fine-tuning (SFT) of the base model.

Results

REFINE-AF consistently outperforms the Self-Instruct baseline across various instruction set sizes on the SUPER-NI benchmark. The improvement holds for all three base models (LLaMA 2-7B, LLaMA 2-13B, Mistral 7B), indicating scalability.

In 10 out of 12 task categories, REFINE-AF achieves higher average ROUGE-L scores. Human evaluation further confirms that models trained with REFINE-AF generate more valid and satisfactory responses with fewer irrelevant outputs.

Increasing the number of instructions gradually improves performance, suggesting that larger datasets from this pipeline yield better alignment.

Significance

REFINE-AF demonstrates that automated feedback can effectively replace human annotation in the instruction generation loop, reducing cost while maintaining or improving quality. The framework is task-agnostic and model-agnostic, making it broadly applicable. By combining RL-based alignment with iterative bootstrapping, it offers a practical recipe for scaling instruction datasets. Future work could explore extending the reward model to handle multimodal inputs or further reducing the seed set size.