Dynamic Layer Routing in LLMs

Analysis

Why This Paper Matters

Large Language Models (LLMs) have become ubiquitous, but their standard practice of processing every token through all layers leads to significant computational waste. Simple queries receive the same depth of processing as complex reasoning tasks, while harder problems may still lack sufficient depth. Prior adaptive-depth methods often required costly inference-time search, architectural changes, or large-scale retraining, and frequently traded accuracy for efficiency gains. Dr. LLM addresses this gap by introducing a retrofittable framework that can be applied to frozen pretrained models, preserving their original weights while enabling dynamic layer routing. This is particularly important for practitioners who need to deploy existing LLMs more efficiently without the expense of retraining or modifying the base architecture.

The paper's use of Monte Carlo Tree Search (MCTS) to derive high-quality layer configurations is a key innovation. MCTS provides explicit supervision for training lightweight routers, ensuring that routing decisions are both accuracy-preserving and compute-budget-aware. This approach avoids the common pitfall of accuracy degradation seen in prior methods, as evidenced by the +3.4%p improvement on ARC and DART benchmarks. The ability to save 5 layers per example on average while improving accuracy is a strong practical result.

Technical Contributions

• Dynamic Layer Routing Framework: Dr. LLM adds a lightweight router to each transformer layer, with three actions: skip, execute, or repeat. This allows the model to adapt its depth per token.
• MCTS-Based Supervision: Monte Carlo Tree Search is used to explore the space of layer configurations under a compute budget, producing optimal or near-optimal routing decisions that serve as training targets for the routers.
• Robust Router Design: The routers use windowed pooling to stabilize routing over long sequences, focal loss with class balancing to handle the inherent imbalance in routing actions, and bottleneck MLP architectures to keep overhead minimal.
• Retrofitting Frozen Models: The framework does not modify the base LLM weights, making it compatible with any pretrained transformer and avoiding the need for large-scale retraining.

Results

Dr. LLM achieves concrete improvements on key benchmarks:

• ARC (logic) and DART (math): Up to +3.4%p accuracy improvement while saving 5 layers per example on average.
• Out-of-domain generalization: On MMLU, GSM8k, AIME, TruthfulQA, SQuADv2, GPQA, PIQA, and AGIEval, accuracy drops only 0.85% while retaining efficiency gains.
• Comparison to prior routing methods: Outperforms existing approaches by up to +7.7%p, demonstrating the effectiveness of explicit MCTS supervision.

These results show that Dr. LLM not only maintains but can improve accuracy while reducing computational cost, a rare combination in adaptive-depth literature.

Significance

Dr. LLM has broad implications for efficient LLM deployment. By enabling dynamic layer routing on frozen models, it offers a practical, low-cost solution for organizations that need to run large models on limited hardware or reduce latency in production. The framework's generalizability to out-of-domain tasks suggests that the learned routing policies capture fundamental properties of layer utility, not just dataset-specific patterns. This work opens the door to further research on retrofitting existing models with lightweight control mechanisms, potentially extending to other forms of dynamic computation such as token pruning or attention head selection. The code release further accelerates adoption and reproducibility.