Prompt Baking: Embedding LLM Behavior in Weights

The document introduces "Prompt Baking," a novel technique for Large Language Models (LLMs) that transforms explicit prompts into permanent updates within the model's weights. Unlike traditional prompting, which is temporary, or fine-tuning, which is data-intensive, Prompt Baking minimizes the difference between a prompted model and an unprompted, "baked" one, achieving comparable performance in minutes. This method effectively alleviates prompt decay over long sequences and enables continuous scaling of prompt strength through "half-baking" or "re-prompting" for enhanced results. The research also explores baking in new knowledge and chain-of-thought examples, demonstrating its resistance to catastrophic forgetting and potential for iterative self-improvement via "Prompt Pursuit."