In-Context Learning: Implicit Weight Dynamics

This academic paper explores In-Context Learning (ICL) in Large Language Models (LLMs), a phenomenon where models learn new patterns from prompts without explicit weight updates. The authors propose that a transformer block implicitly modifies its internal weights during inference, specifically the Multi-Layer Perceptron (MLP) layer, as context is consumed. They introduce the concept of a "contextual block" to generalize this mechanism, demonstrating theoretically and experimentally that context is transformed into a low-rank weight update for the MLP. This work suggests that ICL behaves like an implicit gradient descent, with each token influencing the model's effective weights, providing insights into the mysterious emergent properties of LLMs.