Startups challenge transformer dominance in large language models

The large language model sector is currently dominated by the transformer architecture, which powers most mainstream AI systems but is associated with significant computational costs, high energy consumption, and context limitations. In response, a wave of AI startups—including Subquadratic, Manifest AI, Liquid AI, Inception, and Pathway—is developing alternative architectures. These innovations, such as sparse attention, power retention, liquid neural networks, diffusion-based text generation, and state space models, aim to create faster, more efficient, and potentially more capable AI systems. This shift occurs against a backdrop of rising infrastructure demands, with OpenAI projected to spend $50 billion on computing this year and the International Energy Agency predicting data centre electricity consumption will double by 2030.

The core issue lies in the transformer’s dense attention mechanism, which requires comparing every word in a text block with every other word. As text length increases, the computational load grows exponentially, creating a bottleneck for models designed to process vast amounts of data simultaneously. To address this, Subquadratic, a Miami-based startup, claims to have developed the first sparse attention mechanism that rivals top mainstream models on tasks such as search and coding. Its SubQ model determines word relevance on the fly, significantly reducing the number of calculations required compared to traditional dense attention methods.

Manifest AI, based in San Francisco, is tackling the problem through a mechanism it calls power retention. Rather than processing all tokens, this approach stores only the most relevant information via a rolling summary, preventing the context window from becoming unmanageable. The company has demonstrated this capability with its PowerCoder and Brumby models, claiming the latter rivals Alibaba’s popular open-source Qwen model. This method allows models to handle tasks involving huge datasets, such as analysing hours-long videos or maintaining long-term agent tasks, without the computational overhead of retaining every detail.

Liquid AI, an MIT spinout from Cambridge, Massachusetts, has created Liquid Foundation Models (LFMs), which are hybrid systems comprising 20% transformers and 80% liquid neural networks. Inspired by worm brains, these networks can adapt their behaviour as they learn, unlike fixed transformer models. The resulting models are far smaller and more energy-efficient, capable of running on low-powered devices like a Raspberry Pi. Liquid AI reports nearly 34 million downloads and offers its models for free to organisations with annual revenue under $10 million, matching the performance of rivals four times its size.

Inception, based in Palo Alto, is applying diffusion techniques, typically used for image generation, to text production. By generating text in whole blocks at once rather than word by word, the company claims its Mercury 2 model is 10 times faster than GPT-4 with comparable performance. Meanwhile, Pathway, also in Palo Alto, has built Dragon Hatchling, an LLM using state space models instead of attention mechanisms. This architecture allows for abstract reasoning beyond language sequences, evidenced by its high scores on difficult sudoku puzzles, outperforming several leading LLMs that failed to solve any.

These developments signal a potential transition in how AI systems are constructed. While transformers remain the industry standard, the growing energy costs and contextual limitations are driving innovation toward more efficient and capable alternatives. The success of these startups will determine whether the industry can move beyond the current architectural constraints to meet the escalating demands of advanced artificial intelligence applications.