More and more signs that everyone rushed too early to drown LLM developer problems with hardware. https://techcrunch.com/2025/02/05/researchers-created-an-open-rival-to-openais-o1-reasoning-model-for-under-50/

• Development of the s1 model: Researchers from Stanford and the University of Washington created an AI model for reasoning tasks, called s1. Training the model cost less than $50 in cloud credits. • Training method: The s1 model was obtained via distillation — a process where “specialized” abilities of the original model are transferred to another model. In this case, using supervised fine-tuning (SFT), researchers used answers and the reasoning traces provided by the experimental Google Gemini 2.0 Flash Thinking Experimental model. • Base model and data: An existing free model from the Chinese lab Qwen (owned by Alibaba) was used as the base. For training, a relatively small dataset of 1,000 carefully selected questions with answers and explanations was created. • Results and efficiency: Training took less than 30 minutes using 16 Nvidia H100 GPUs, and the resulting model showed performance comparable to leading models like OpenAI o1 and DeepSeek R1, especially on math and programming tasks. A technique was also used where the model asked to “wait,” allowing it to double-check its answers and improve accuracy. • Significance and consequences: Building such a model at a modest cost demonstrates that competitive AI solutions can be created without large financial investments, raising questions about standards and IP protection in AI. At the same time, large labs like OpenAI expressed dissatisfaction with the data extraction methods used for such projects. • Future outlook: Despite the success of distillation as a method for inexpensive reproduction of existing model capabilities, significant investments may still be required to create fundamentally new breakthrough AI systems, as companies like Meta, Google, and Microsoft plan to do.