The Billion-Vector Problem: HNSW vs. DiskANN in Azure AI Search

Most architects default to HNSW because it's the industry standard. It's the algorithm used by most vector databases, the one featured in tutorials, and the option many teams deploy without a second thought.For small and medium-sized workloads, that's often the right decision.But at enterprise scale, a hidden problem begins to emerge.The moment organizations start dealing with hundreds of millions—or even billions—of embeddings, the economics of vector search change dramatically. What looked like a straightforward architectural decision suddenly becomes a conversation about infrastructure budgets, memory consumption, scalability, and long-term sustainability.In this episode of the M365 FM Podcast, we explore one of the most important design decisions facing enterprise AI architects today: when should you use HNSW, and when does DiskANN become the better option?More importantly, we examine how this decision impacts Azure AI Search, Azure Cosmos DB, Microsoft 365 Copilot-style architectures, Retrieval-Augmented Generation (RAG) systems, and the future of large-scale enterprise search.
WHY VECTOR SEARCH CHANGES EVERYTHING

Traditional search systems rely on keywords. They look for exact matches between a query and the words stored inside documents. While this approach works reasonably well for structured content, it struggles when users describe concepts differently than the documents themselves.Vector search solves this challenge by converting both documents and queries into embeddings—high-dimensional numerical representations of meaning. Instead of searching for matching words, vector databases search for semantic similarity.This is the foundation of modern AI-powered search experiences, enterprise copilots, and Retrieval-Augmented Generation systems. It allows users to find information based on intent rather than exact terminology, dramatically improving discovery across large knowledge repositories.
THE REAL CHALLENGE ISN'T SEARCH—IT'S SCALE

Most conversations about vector search focus on retrieval quality, embeddings, and similarity algorithms.Far fewer discussions focus on the infrastructure required to make those searches happen.Every vector must be stored somewhere. Every nearest-neighbor calculation requires an index. Every index consumes resources.At smaller scales, those requirements are manageable.At enterprise scale, they become the dominant factor in architectural decisions.The episode explores how the physical location of your vector index—whether it lives entirely in memory or partially on disk—ultimately determines the economics of large-scale AI systems. This seemingly technical distinction becomes one of the most important variables affecting cloud costs, scalability, and long-term platform viability.
UNDERSTANDING HNSW

Hierarchical Navigable Small World (HNSW) has become the gold standard for approximate nearest neighbor search.The algorithm uses a sophisticated graph structure that enables extremely fast vector retrieval with impressive recall rates. By organizing vectors into interconnected layers, HNSW can navigate large vector spaces with remarkable efficiency.Its strengths are easy to understand:
• Extremely low latency
• Excellent recall quality
• Mature ecosystem support
• Broad industry adoptionFor small and medium-sized vector workloads, HNSW remains one of the best options available.However, the algorithm is built around a critical assumption: the entire graph must remain in memory.That assumption becomes increasingly expensive as datasets grow. What begins as a performance advantage eventually becomes a scalability challenge, particularly when organizations move into the hundreds of millions of vectors.
THE HNSW MEMORY WALL

One of the most eye-opening discussions in this episode focuses on what happens when vector indexes reach massive scale.Memory consumption grows alongside the graph, and eventually organizations encounter what many architects now call the memory wall.At this point, infrastructure requirements shift from ordinary compute resources to specialized memory-optimized environments. Replication, disaster recovery, regional deployments, and high-availability architectures multiply those requirements even further.The result is that an algorithm originally selected for performance can eventually become one of the largest cost drivers within an AI platform.This isn't a failure of HNSW.It's simply a consequence of the architectural assumptions that made HNSW successful in the first place.
ENTER DISKANN

DiskANN was developed by Microsoft Research to address the scaling limitations associated with memory-heavy vector search architectures.Rather than keeping the entire graph in RAM, DiskANN uses a hybrid approach that combines memory-resident navigation structures with SSD-based storage for full-precision verification.The result is a system capable of maintaining high retrieval quality while dramatically reducing memory requirements.This architectural shif...