The week on AI – November 17, 2024

Are LLM reaching a plateau?

The reasoning capabilities of LLMs may be reaching a plateau, suggesting that the scaling laws might be hitting a limit. Scaling laws which are based on observations and are not proper laws (like the Moore Law), describe how machine learning models improve as a function of resource allocation, such as compute power, dataset size, or model parameters. Reports suggest that OpenAI’s upcoming model, Orion, is showing only modest improvements over GPT-4, falling short of the significant leaps seen in earlier model iterations. The industry is beginning to exhaust its data for training LLMs, and the legal disputes over copyright rights are escalating. Additionally, the use of synthetic data generated by AI presents its own set of challenges. In addition, computation power is not limitless, even in the cloud, and it brings limitations and hard decisions for LLM developers like OpenAI. The industry is working to overcome these challenges by developing new training approaches that align more closely with human thinking. This has already been used in the development of OpenAI’s o1 model.

Google DeepMind has a new way to look inside AI models

As previously discussed, we currently do not fully understand how AI operates. Google DeepMind has taken on this challenge by introducing Gemma Scope, which is a collection of open, sparse autoencoders (SAEs) aimed at providing insights into the internal workings of language models. This research falls under the category of mechanistic interpretability. To better control AI, we will need to further refine our approaches, balancing the need to reduce or eliminate undesirable behaviors—like promoting violence — without compromising the model’s overall knowledge. Additionally, removing undesirable knowledge is a complex task, particularly when it involves information that should not be widely disseminated (such as bomb-making instructions) as well as knowledge that may be incorrect [on the internet]. Mechanistic interpretability has the potential to enhance our understanding of AI, ensuring that it is both safe and beneficial. Read

Elevating AI-coding to the next level

In a crowded landscape filled with AI coding tools such as GitHub Copilot, Dodeium, Replit, and Tabnine, many of these options function primarily as coding assistants. Tessl aims to elevate AI-based coding to the next level. They envisions a future where software developers transition into roles more akin to architects or product managers, allowing artificial intelligence to handle the majority of the coding. Upon examining their proposal on their website, it seems that Tessl is not attempting to turn everyone into a developer (at least not yet). Their tool will still be targeted at developers but will empower them to define what they want to build and let the Tessl AI tool define the internal architecture of the solution and develop it. Let’s see how far they can push the concept. They have just raised another USD 100 million making them worth a reported USD 750 million. Read

Other readings

> Inside Elon Musk’s colossus supercomputer, watch (no content guarantee)
> Amazon to develop its own AI chips to take on Nvidia, read
> Nvidia’s message to global chipmakers, read
> A.I. Chatbots Defeated Doctors at Diagnosing Illness, Read