RAG systems are often over-engineered, says new analysis

Retrieval-Augmented Generation (RAG) systems are frequently more complex than necessary, according to a new article published in the Lighthouse Newsletter. The piece challenges the prevailing industry trend of immediately adopting embeddings, vector databases, and reranking pipelines, suggesting instead that most developers are over-engineering their stacks. The author argues that for the majority of applications, basic methods such as BM25 full-text search and large language model (LLM) based query rewriting are sufficient to meet user needs.

The article compares six distinct retrieval architectures, ranging from naive full-text search to advanced agentic systems. It proposes a decision framework based on five key factors: data freshness requirements, corpus characteristics, query patterns, scale and performance, and team capabilities. The analysis suggests that developers should select the simplest approach that meets their specific requirements, reserving more complex embedding strategies for cases where specific data or performance demands justify the added overhead.

According to the Lighthouse Newsletter, approximately 60 per cent of systems should stop at full-text search combined with query rewriting. Another 25 per cent require hybrid approaches, such as on-the-fly or hot/cold tiering, while only 10 per cent need full pre-embedding and 5 per cent require custom solutions. The author notes that basic methods like BM25 handle a significant portion of use cases without the complexity of vector databases, particularly when users write keyword-style queries or when exact matches are required.

The article highlights that LLM-based query rewriting can address many semantic search problems by transforming messy user inputs into clean keyword searches. This process costs approximately $0.001 per query when using models like GPT-4o-mini and allows for rapid iteration without re-embedding the entire corpus. In contrast, embedding strategies introduce challenges such as determining chunk sizes, managing model deprecation, and handling increased latency, with on-the-fly embedding adding 200 to 500 milliseconds per query.

For teams without dedicated machine learning expertise, the analysis advises staying with full-text search plus query rewriting. It notes that if results are unsatisfactory, adjusting the system prompt for query rewriting is far simpler than re-evaluating chunking strategies or running regression tests on new embeddings. The article emphasises that for proprietary terminology, exact keyword matching often outperforms general-purpose semantic understanding, making simple tools more effective for specific domains.

The Lighthouse Newsletter concludes that developers should measure their baseline performance before increasing complexity. If users are satisfied with current results, no further optimisation is needed. If issues arise, query rewriting should be tested first due to its low cost and zero re-indexing requirements. Only when data proves that simpler approaches are insufficient should teams move to hybrid search or full pre-embedding, ensuring they do not build a solution for a 5 per cent problem when a 60 per cent solution would suffice.