Gemma APS

Analysis

Why This Paper Matters

Abstractive proposition segmentation—breaking text into atomic, self-contained facts—is crucial for many NLP applications, from fact-checking to summarization. However, existing methods often rely on rule-based or unsupervised approaches that lack accuracy or scalability. This paper tackles the challenge head-on by framing proposition segmentation as a supervised learning task, leveraging large language models (LLMs) and synthetic data to build a domain-general solution. The key insight is that by training a compact student model on diverse synthetic data generated by powerful teachers, one can achieve both high accuracy and broad domain coverage without requiring expensive human annotations for every new domain.

How It Works

The authors first define the task precisely: given an input text, the model must output a list of propositions that are well-formed, atomic, self-contained, supported by the text, and comprehensive. They explore two output formats: ungrouped propositions (a flat list) and grouped propositions, where propositions are organized by the original sentences using special tokens.

For evaluation, they introduce two families of metrics based on Natural Language Inference (NLI). Reference-free metrics check whether predicted propositions are supported by the input text (precision) and whether all facts in the text are covered (recall). Reference-based metrics compare predicted propositions against a gold standard, using bidirectional NLI to ensure atomicity and self-containedness.

The core methodology involves a three-step distillation pipeline. First, teacher LLMs (Gemini Pro and Gemini Ultra) are fine-tuned on the ROSE dataset, which contains news summaries annotated with atomic content units. Second, a large multi-domain synthetic dataset is generated by prompting FLAN-PaLM2 and Gemini Ultra to produce texts across 75 manually selected domains, with two lengths (short and paragraph). This yields 226K examples. Third, the teachers are applied to this synthetic corpus to produce grouped proposition annotations, and a Gemma 7B model is trained on these pairs.

Results

The authors evaluate on three datasets: ROSE (in-domain), Reddit, and Amazon Review (out-of-domain). Key findings include:

• Grouped proposition formats consistently outperform ungrouped ones.
• The distilled student model achieves reference-free metrics ≥96% on all datasets, indicating strong support and comprehensiveness.
• On out-of-domain datasets, the student model significantly outperforms a Gemma 7B model trained directly on ROSE, demonstrating the effectiveness of synthetic data for domain adaptation.
• The student model performs comparably to or better than the teacher models on out-of-domain data, while being much smaller and more efficient.
• Few-shot prompting baselines show high precision but low recall (53-83% reference-free), making them unreliable for tasks requiring complete fact coverage.

Significance

This paper presents a practical, scalable solution for proposition segmentation that generalizes across domains without requiring new annotations. The distillation approach reduces the computational cost of using large LLMs while maintaining high accuracy. The grouped proposition format also provides automatic sentence attribution, which is valuable for downstream tasks like grounding and fact verification. The introduced evaluation metrics offer a principled way to assess proposition quality, and the released models on HuggingFace enable immediate use by the community. This work moves proposition segmentation from a niche research problem to a deployable tool for real-world NLP systems.