TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward

Analysis

Why This Paper Matters

Compositional text-to-image generation remains a critical challenge for diffusion models, which often fail to faithfully render all concepts in complex prompts (e.g., "a red cube on top of a blue sphere"). Existing solutions typically require retraining, external reward models, or complex prompt engineering. TILT addresses this gap with a training-free, test-time method that leverages the model's own internal distributions to guide sampling toward joint concept presence. This is significant because it avoids the cost and brittleness of external supervision, making compositional improvement accessible to any pre-trained diffusion model.

The paper's framing of compositional failures as overlap modes between joint and single-concept distributions is a novel perspective that leads to a principled reward formulation. By deriving a closed-form tilted target distribution, TILT provides mathematically grounded guidance steps that are easy to integrate into existing sampling pipelines. This could democratize high-quality compositional generation for practitioners who cannot afford fine-tuning or external models.

Technical Contributions

• Model-Intrinsic Reward: Defines a reward that favors samples where all concepts are jointly present, computed solely from the base model's own probability distributions without external classifiers or reward models.
• KL-Constrained Objective: Formulates a KL-constrained optimization problem that yields a closed-form tilted target distribution, enabling principled guidance during diffusion sampling.
• Two Guidance Strategies: The interaction of concept distributions naturally leads to two distinct guidance strategies—one focusing on joint probability and another on conditional probability—each with different trade-offs.
• Hybrid Approach: A hybrid strategy that balances the two guidance methods achieves stronger performance by combining their respective benefits.
• Training-Free: The entire framework operates at test time, requiring no additional training or fine-tuning of the base diffusion model.

Results

TILT was evaluated on prompts from T2ICompBench, a standard benchmark for compositional generation. The method improves compositional alignment compared to previous baselines (e.g., Attend-and-Excite, Composable Diffusion) while preserving image quality metrics such as FID and CLIP score. The hybrid guidance strategy consistently outperforms individual strategies, demonstrating the value of balancing joint and conditional guidance. Quantitative results show gains in concept coverage and reduction in missing-object errors without sacrificing visual fidelity.

Significance

TILT advances the field by providing a theoretically grounded, training-free solution to a persistent problem in text-to-image generation. Its reliance on model-intrinsic rewards means it can be applied to any diffusion model without external dependencies, making it broadly useful for practitioners. The work also opens avenues for further research into intrinsic reward design and test-time alignment in generative models. By improving compositional fidelity, TILT could enhance the reliability of AI-generated images in applications like design, education, and content creation.