KL Divergence VS MSE for Knowledge Distillation

Analysis

Why This Paper Matters

Knowledge distillation (KD) is a cornerstone technique for compressing large neural networks into smaller, deployable models. Traditionally, KD uses KL divergence between softened teacher and student probability distributions, controlled by a temperature hyperparameter τ. This paper challenges that convention by demonstrating that a simple mean squared error (MSE) loss between raw logits—bypassing softmax entirely—often yields better results. The finding is significant because it simplifies the distillation pipeline, reduces hyperparameter sensitivity, and provides theoretical insight into why logit matching outperforms probability matching. Additionally, the paper addresses practical concerns like noisy teachers and label noise, offering actionable strategies for real-world deployment.

How It Works

The authors first analyze KL divergence mathematically. As τ increases, the gradient of KL divergence increasingly forces the student's logits to match the teacher's logits (logit matching). As τ approaches zero, the gradient focuses only on whether the student's predicted class matches the teacher's (label matching). Empirical results show that logit matching correlates with better generalization, motivating the use of MSE directly on logits: L_MSE = ||z_s - z_t||^2, where z_s and z_t are student and teacher logits.

Experiments on CIFAR-100 with Wide-ResNet (WRN) and ImageNet with ResNet (RN) families compare L_KL (with varying τ) and L_MSE. The authors measure logit distances and penultimate layer representations. They find that L_MSE produces student logits closer to the teacher's than L_KL even with large τ, and that L_MSE shrinks the student's pre-logit representations more tightly to the teacher's.

Grid maps of accuracy versus α (weight on soft targets) and τ show that increasing α and τ both improve performance, confirming that soft targets are more informative. When α=1.0, larger τ yields lower generalization error.

Results

Direct comparison shows L_MSE outperforms L_KL across various teacher-student pairs. For example, on CIFAR-100 with WRN-28-4 teacher and WRN-16-2 student, L_MSE achieves higher top-1 accuracy than L_KL with any τ. L_MSE also matches or exceeds existing KD methods.

Logit distribution analysis confirms L_MSE yields tighter alignment. Penultimate layer visualizations show L_MSE forces student representations to follow the teacher's shape more cohesively.

For noisy teachers (poorly trained), label matching (L_KL with small τ) is more robust than logit matching. L_KL with τ≤1.0 achieves best test accuracy under label noise, as it avoids transferring overconfidence.

Sequential distillation (large→medium→small) is not always beneficial; direct medium→small distillation often works better. However, when capacity gaps are large, sequential steps can help.

Significance

This paper reframes knowledge distillation as a logit regression problem rather than a probability matching one. The practical implication is clear: practitioners can replace KL divergence with MSE loss for simpler, often better distillation. The analysis of temperature effects provides a principled way to handle noisy labels—using small τ in KL divergence acts as a form of label smoothing. The sequential distillation findings offer guidance for model compression pipelines. Overall, this work reduces the engineering overhead of KD while improving performance, making it more accessible for deployment in production systems.