In-Context Learning: Implicit Weight Dynamics

This academic paper explores In-Context Learning (ICL) in Large Language Models (LLMs), a phenomenon where models learn new patterns from prompts without explicit weight updates. The authors propose that a transformer block implicitly modifies its internal weights during inference, specifically the MLP layer, based on the provided context. Through theoretical analysis and experimentation, they demonstrate that this implicit modification can be represented as a low-rank weight update, which acts similarly to a fine-tuning process. Furthermore, the paper establishes that this implicit learning process exhibits dynamics akin to stochastic gradient descent, where each context token contributes to a weight adjustment that minimizes a changing loss function. This research offers a more generalized understanding of ICL, extending beyond previous works that relied on more restrictive assumptions about self-attention layers.