Selective Self-to-Supervised Fine-Tuning (S3FT)

Analysis

Why This Paper Matters

Standard supervised fine-tuning (SFT) of large language models often suffers from overfitting, especially when the training data contains only one gold response per input, ignoring the fact that many inputs have multiple valid outputs. This paper introduces Selective Self-to-Supervised Fine-Tuning (S3FT), a novel approach that addresses this limitation by selectively using the model's own correct predictions as training targets. This is significant because it leverages the model's own knowledge to regularize training, reducing overfitting and improving generalization without requiring additional data or architectural changes.

The paper matters because it tackles a fundamental issue in LLM fine-tuning: the mismatch between the one-to-one mapping in standard SFT and the one-to-many nature of real-world language tasks. By allowing the model to learn from its own correct outputs, S3FT effectively increases the diversity of training signals and encourages the model to maintain its pre-trained capabilities. This is particularly relevant for practitioners who need to adapt LLMs to specific domains with limited data, where overfitting is a major concern.

Technical Contributions

• Selective training target selection: S3FT dynamically chooses between the model's own output and the gold response based on alignment, avoiding forcing the model to memorize a single correct answer.
• Self-consistency regularization: By fine-tuning on its own correct predictions, the model reinforces its existing knowledge, acting as a form of self-distillation.
• Paraphrase fallback: When the model's output does not match the gold response, S3FT uses the gold response or a paraphrase, ensuring the model still learns from high-quality targets.
• Simplicity and compatibility: The method requires no changes to the model architecture or training pipeline, making it easy to integrate into existing SFT workflows.

Results

The abstract reports that S3FT improves upon standard SFT by mitigating overfitting and improving generalization performance. While specific numerical metrics are not provided in the abstract, the key result is that selective fine-tuning on self-generated correct outputs leads to better performance on unseen data compared to standard SFT. This suggests that the method effectively balances memorization and generalization.

Significance

S3FT has broad implications for the AI field, particularly in fine-tuning LLMs for downstream tasks. It provides a principled way to incorporate the idea that multiple valid responses exist for a given input, which is common in open-ended generation, dialogue, and creative tasks. This could lead to more robust and adaptable models that retain their pre-trained knowledge while learning new tasks. For practitioners, S3FT offers a low-cost improvement over standard SFT that can be easily adopted, potentially becoming a default fine-tuning strategy in many applications.