Malika Aubakirova and Matt Bornstein — a16z - Why We Need Continual Learning - April 2026
FreeWhy LLMs need to learn continuously, not just in context
FreeFree tier
About Malika Aubakirova and Matt Bornstein — a16z - Why We Need Continual Learning - April 2026
An article by Malika Aubakirova and Matt Bornstein, published on a16z's Substack, arguing for the importance of continual learning in AI. Using the film Memento as an analogy, the authors explain that current large language models suffer from a 'perpetual present'—they cannot update their parameters post-deployment. The piece distinguishes between in-context learning (ICL) and parametric learning, advocating for research into compressed, weight-updating memory architectures. It synthesizes insights from leading researchers and positions continual learning as a critical next step for scaling AI capabilities beyond static knowledge.
Key Features
Uses Memento film analogy to illustrate static parameter problem in LLMs
Distinguishes between in-context learning and parametric (weight-updating) learning
Makes case for continual learning as a new dimension of scaling
Draws on conversations with top researchers, PhD students, and startup founders
Discusses limitations of ICL for novel mathematics, security, and tacit knowledge
Advocates for models that learn their own memory architectures
Pros & Cons
Pros
- Engaging, accessible analogy (Memento) makes a complex concept relatable
- Well-researched, incorporating insights from experts in the field
- Clearly defines and differentiates key terms (ICL vs. parametric learning)
- Provides a strong motivation for new research and startup efforts
- Connects theoretical need with practical implications for scaling
Cons
- Does not include specific technical details or implementation examples
- Offers no concrete algorithms or code for continual learning
- Opinion piece focused on 'why' rather than 'how'
- Assumes reader familiarity with LLM training and inference basics
Best For
Understanding the fundamental limitations of current LLM architecturesGuiding research directions in lifelong learning and memory-augmented modelsInforming startup strategy in the continual learning ecosystemEducating technical leaders on the need for post-deployment model updatesProviding a high-level overview for investors evaluating AI research
FAQ
What is continual learning in the context of this article?
Continual learning refers to the ability of AI models to update their parameters (weights) after deployment, compressing new experiences directly into the model rather than relying solely on in-context learning.
Why does the article compare LLMs to the film Memento?
Because LLMs, like the protagonist in Memento, cannot form new memories. They have vast static knowledge but reset with each new context, requiring external scaffolding (chat history, retrieval systems) to function.
What problems does the article claim in-context learning cannot solve?
Problems requiring genuine discovery (novel mathematics), adversarial scenarios (security), and tacit knowledge too complex to express in language are cited as areas where ICL is insufficient.