OLMoE

professor norris: Welcome back to Mechanical Dreams, the podcast where we delve into the exciting world of machine learning and natural language processing. I'm Professor Norris, and as always, I'm joined by my brilliant student, Linda.

linda: It's great to be back, Professor. And I'm particularly excited about today's paper. It tackles a topic that's been buzzing in the NLP community: Mixture-of-Experts models, or MoEs for short.

professor norris: Ah yes, MoEs. I remember when they were a promising but somewhat fringe concept. It seems they're making a comeback, especially with industry giants like Google incorporating them into their frontier models.

linda: Exactly! And that's what makes today's paper so intriguing. It's not just about pushing the boundaries of MoE performance but also about making this technology accessible to the wider research community. The paper is titled "OLMOE: Open Mixture-of-Experts Language Models."

professor norris: Open, you say? That's certainly a welcome change in a field often dominated by closed-source, proprietary models. What makes OLMOE so open, Linda?

linda: Well, Professor, the authors have gone above and beyond the usual practice of just releasing model weights. They've open-sourced everything: the model weights, the training data, the code, and even the training logs.

professor norris: That's remarkable! Such transparency is crucial for advancing our understanding of MoEs, which, as you know, introduce a whole new layer of complexity to language modeling. Tell me, Linda, what are some of the key design decisions involved in building a successful MoE model?

linda: That's a great question, Professor. One of the primary decisions is determining the number of experts and how many of those experts are activated for each input.  There's also the question of expert granularity: should we use a few large experts or many smaller ones? And then there's the routing algorithm, which decides how to assign inputs to the appropriate experts.

professor norris: These are indeed crucial decisions. And if I recall correctly, there's also the matter of whether to share experts across layers, right?

linda: Absolutely, Professor. That's another important design choice that can significantly impact performance.

professor norris: So, how does OLMOE approach these design challenges, Linda?

linda: OLMOE-1B-7B, the specific model they focus on, has a total of 7 billion parameters, but only 1.3 billion are active for each input. This makes it comparable to dense models with around 1 billion parameters in terms of inference cost.

professor norris: That's clever. They're essentially trying to achieve the efficiency of a smaller model while leveraging the capacity of a much larger one.

linda: Precisely! And they've opted for a fine-grained approach with 64 small experts per layer, out of which 8 are activated. They use a token-based routing algorithm called "dropless" to assign inputs to experts.

professor norris: And do they share experts across layers, Linda?

linda: No, Professor. They found that sharing experts didn't provide any significant benefits.

professor norris: Interesting. So, how well does OLMOE perform compared to other models, both dense and MoE?

linda: Well, it significantly outperforms all open 1-billion parameter models. It even achieves competitive performance on common benchmarks like MMLU compared to dense models with significantly higher inference costs, like the Llama2-13B.

professor norris: That's quite impressive! And what about after adaptation with instruction tuning and preference tuning?

linda: They create OLMOE-1B-7B-INSTRUCT, which further improves performance and even exceeds larger instruct models, including the Llama2-13B-Chat and DeepSeekMoE-16B, on various benchmarks.

professor norris: Remarkable! It