Trends in Open-Source Language Models

AI progress has long been dominated by raw scale—larger datasets, bigger models, and massive compute budgets. But recent breakthroughs suggest that efficiency in training, retrieval, and reasoning may now be more important than brute force scaling. The first shock came with DeepSeek-R1, an open-source model that demonstrated that reinforcement learning (RL) alone—without extensive supervised fine-tuning—can develop reasoning capabilities comparable to proprietary models [1]. This shift is reinforced by Qwen 2.5’s architecture optimizations and Janus-Pro’s multimodal advancements, proving that cheaper, faster, and more effective AI is possible without simply increasing parameter counts [2]. DeepSeek-R1 shows that RL can be a primary mechanism for improving LLM reasoning, not just an alignment tool [1]. Its initial version, DeepSeek-R1-Zero, trained purely via RL, displayed strong reasoning but suffered from readability issues. The refined DeepSeek-R1, incorporating minimal cold-start data and rejection sampling fine-tuning, reached OpenAI-o1-1217-level performance at a fraction of the cost. This challenges the conventional pretraining-heavy paradigm. AI architecture is also undergoing a fundamental shift. Janus-Pro, from DeepSeek-AI, introduces a decoupled approach to multimodal AI, separating image understanding from image generation [2]. Unlike previous models that forced both tasks through a shared transformer, Janus-Pro optimizes each independently, outperforming DALL-E 3 and Stable Diffusion 3 Medium in instruction-following image generation. At a more fundamental level, Bytedance’s Over-Tokenized Transformers reveal a silent inefficiency in LLM design: tokenization is a bottleneck [3]. Their research shows that expanding input vocabulary—while keeping output vocabulary manageable—drastically reduces training costs and improves performance. A 400M parameter model with an optimized tokenizer matched the efficiency of a 1B parameter baseline (!), proving that many LLMs are computationally bloated due to suboptimal tokenization strategies. Beyond efficiency, AI is also becoming more structured in reasoning and retrieval. Google DeepMind’s Mind Evolution introduces a genetic algorithm-like refinement process [4], evolving multiple solution candidates in parallel and iteratively improving them. This could lead to AI systems that autonomously refine their own answers rather than relying on static generation. Meanwhile, Microsoft’s CoRAG is redefining RAG by solving the multi-hop retrieval challenge [5]. Standard RAG models retrieve once before generating a response, failing on multi-step queries. CoRAG introduces recursive retrieval, dynamically reformulating queries at each step, leading to a 10+ point improvement on multi-hop QA benchmarks. The combined effect of these breakthroughs is a shift in how AI is trained, how it retrieves knowledge, and how it reasons in real time - everything you need to design more intelligent brains.

The frenzy around the new open-source reasoning #LLM, DeepSeek-R1, continued today, and it’s no wonder. With model costs expected to come in 90-95% lower than OpenAI o1, the news has reverberated across the industry from infrastructure players to hyperscalers and sent stocks dropping. Amid the swirl of opinions and conjecture, I put together a brief synopsis of the news – just the brass tacks – to try and simplify the implications and potential disruptions and why they matter to leaders. 1. 𝗦𝗸𝗶𝗽𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗥𝘂𝗹𝗲𝘀: DeepSeek-R1-Zero ditched supervised fine-tuning and relied solely on reinforcement learning—resulting in groundbreaking reasoning capabilities but less polished text. 2. 𝗧𝗵𝗲 𝗣𝗼𝘄𝗲𝗿 𝗼𝗳 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮: Even a tiny set of curated examples significantly boosted the model's readability and consistency. 3. 𝗦𝗺𝗮𝗹𝗹 𝗕𝘂𝘁 𝗠𝗶𝗴𝗵𝘁𝘆 𝗠𝗼𝗱𝗲𝗹𝘀: Distilled smaller models (1.5B–70B parameters) outperformed much larger ones like GPT-4o, proving size isn’t everything. Why does this matter to business leaders?  • 𝗚𝗮𝗺𝗲-𝗖𝗵𝗮𝗻𝗴𝗲𝗿 𝗳𝗼𝗿 𝗔𝗜 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗖𝗼𝘀𝘁𝘀: Skipping supervised fine-tuning and leveraging reinforcement learning could reduce costs while improving reasoning power in AI models. • 𝗛𝗶𝗴𝗵-𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮 𝗶𝘀 𝗮 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗔𝗱𝘃𝗮𝗻𝘁𝗮𝗴𝗲: Investing in carefully curated data (even in small quantities) can lead to a competitive edge for AI systems. • 𝗦𝗺𝗮𝗹𝗹𝗲𝗿 𝗦𝗺𝗮𝗿𝘁𝗲𝗿 𝗠𝗼𝗱𝗲𝗹𝘀 𝗦𝗮𝘃𝗲 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Smaller, distilled models that perform better than larger ones can drive efficiency, cutting costs on infrastructure while maintaining high performance. Let me know if you agree… And if you're curious, the DeepSeek-R1 paper is a must-read. https://lnkd.in/eYPidAzg #AI #artificialintelligence #OpenAI #Hitachi

💡 Looks like knowledge distillation is becoming a popular trend for building small language models and Gemma 2 has joined in on it! Meta recently announced that their smaller Llama 3.1 models were distilled from the larger 405B model, and they saw good performance. The latest Gemma paper shows they are doing the same thing. ⛳ Some insights from the report: 👉 The authors report that they train their smaller Gemma models with knowledge distillation instead of next token prediction. This approach can reduce the training time of smaller models by giving them richer gradients. 👉 Specifically , they use a large language model as a teacher to train small models, the Gemma 2B and 9B models, on a quantity of tokens that is more than 50 times the compute-optimal quantity predicted by the theory 👉 They observe that the performance gains from distillation remain as the model size is scaled ranging from 5% to 15% By using knowledge distillation and updated attention mechanisms, Gemma 2 achieves competitive performance against much larger models (2-3x the size), showing the potential of these techniques in optimizing small-scale language models for diverse applications.

There has been a tidal wave of big AI news this week -- Stargate, OpenAI + Microsoft. But as the week winds down, I am still thinking about the implications of another big announcement: the release of DeepSeek-R1. Developed by China's DeepSeek AI, it matches or beats OpenAI's o1 across such tasks as math, code, and reasoning tasks. When it comes to math benchmarks, the smallest model outperforms GPT-4o and Claude 3.5 Sonnet. Here's what The Neuron - AI News said about it: "While America's tech giants were busy toasting Trump, a Chinese company just casually released what might be the biggest AI breakthrough since ChatGPT." Wow. Because of the geopolitical situation, there is a lot of focus on the China vs US angle. But in thinking about #GenAI business models, there are two very important elements of the DeepSeek story. First, according to The Neuron: "DeepSeek discovered that pure reinforcement learning enables a language model to automatically learn to think and reflect." Translation: While OpenAI was building a moat around the need for massive data to train AI, DeepSeek may have breached that moat with a trial-and-error method that doesn't require such data. The implications for costs and competition in the LLM space are big. Second, DeepSeek is Open Source. This has been a theme I've been exploring a lot over the past year, and it remains one to closely watch in this space. As Yann LeCun wrote on LinkedIn: "To people who see the performance of DeepSeek and think: 'China is surpassing the US in AI.' You are reading this wrong. The correct reading is: 'Open source models are surpassing proprietary ones.'" The Open vs. Closed dynamic changes the economics in terms of entering the LLM space and maintaining a competitive advantage. Last September I wrote about the LLM competition in an article called "GenAI Foundation Models: The LLM Race Has Only Just Begun." The DeepSeek release re-emphasizes just how open to disruption this market remains. See links in the comments to the articles cited. 👇

AI research community recent paper by Google DeepMind researchers, including Hritik Bansal , Arian Hosseini , Rishabh A. , Vishal M. Patel , and Mehram Kazemi, reveals that training #LargeLanguageModels (#LLMs) with data from smaller, less resource-intensive models can yield superior performance. This approach challenges the conventional wisdom of using larger, more expensive models for fine-tuning. Here's a deep dive into their findings: 🔹 Findings: Models fine-tuned on weaker, cheaper data outperform those trained on stronger data across benchmarks. 🔹 Implications: This could revolutionize how we approach #LLMtraining, making AI more accessible and efficient. 🔹 Key Takeaways: - Compute-optimal sampling from smaller models provides better coverage and diversity. - Efficiency in AI doesn't always require the biggest models. For those interested in AI efficiency and model compression, here are some techniques to consider: 1. #Pruning - Reducing less important weights in neural networks. 2. #Quantization - Lowering the precision of model parameters for efficiency. #ArtificialIntelligence #MachineLearning #DeepLearning #ModelCompression #AIResearch #GoogleDeepMind #HritikBansal #AO

The Rise of Mistral's Open Source LLM 8x7B Recently on the a16z podcast, Arthur Mensch, CEO of Mistral, thought that the gap between closed and open source LLMs might be 6 months. That might have been way off. It may have already closed. Mistral's 8x7B model, which just dropped last month, is now a rising star on the Hugging Face LLM Leaderboard, where AI models are voted on in real-time. It's not a perfect system, but it's a strong indicator of what research might later validate. Debuting last month, 8x7B quickly climbed to the 7th spot on the leaderboard, becoming the only open-source model in the top 10. While it seems that open and closed source LLMs will converge on the text side, there some other factors to consider. First, Multimodality. Excelling in text alone isn't enough. Modern AI models must handle audio, speech, images, and soon, video. ChatGPT does this pretty well, where you can generate funny images and also speak directly with the model. Next, the Application Layer. It's all about user experience. ChatGPT exemplifies this, offering an engaging and user-friendly interface, especially when compared with something like Google Bard, which I’ve personally found frustrating to work with. Lastly, Ecosystem Integration. It's about how these models integrate with our data and teams. ChatGPT's recent team plan launch, promoting sharing and internal custom GPTs, underscores the importance of connecting these models to our data to make them smarter about what we care about. In conclusion, while open source models are rapidly advancing in text capabilities, proprietary models, supported by their companies, are likely refocusing on multimodality, enhanced application layers, and deeper integration with user data and teams. I’m eager to read your thoughts. Please share them below. #genai #mistral #opensourceai

The rate of progress with the use of LLMs for AI has significantly slowed down after the launch of ChatGPT 4, and many have been suspecting that we had reached a saturation point with that approach. My take is that the dramatically better LLMs will require dramatically more resources, both for training and inference, and we are only now getting to the point where that is feasible, particularly for the publicly available models. However, one approach that has emerged over the past year that can still leverage the present day LLMs and squeeze even more juice out of them seems to be the use of inference-time compute. i.e., instead of just using compute to deliver the pre-processed information, compute is also being used at the test-time in order to "reason" through several possible responses. OpenAI has been the pioneer in this regard, with rumors of their "Q*" and "Strawberry" models circulation around for well over a year. A few months ago they finally released o1-mini and o1-preview models with these "reasoning" abilities, and last week we got access to the fully capable o1 and o1-pro. On social media members of the technical staff at OpenAI have voiced their confidence that OpenAI has a substantial and hard-to-catch-up lead with this approach. However, just this week Hugging Face has announced an intreating open source project that can replicate some of the core features of the o1 test-time inference approach. This open source project claims to be able to use smaller models in order to achieve the predictive accuracy of much larger models by repeated processing of the prompted instructions and use of chain of reasoning. I have not tried using their code yet myself, but based on the community reaction it seems to be very viable. If its promises seem to bear out, then it's very likely that the open source AI could overtake the propriety approaches in a very near future, and those of us with lots of compute at home could become even more productive with the use of LLMs and AI in our workflows. Link to the HuggingFace blog post: https://lnkd.in/dDubbyTr