Architecture Blueprint

Design Real-Time Vector Database (HNSW & Product Quantization)

Distributed high-dimensional vector search engine executing Approximate Nearest Neighbor (ANN) similarity queries across hundreds of millions of embeddings with metadata filtering and real-time indexing.

1. System Requirements

Functional

Ingest high-dimensional vector embeddings (768 to 3072 dimensions) with associated JSON metadata.
Execute Approximate Nearest Neighbor (ANN) search using cosine distance, Euclidean distance (L2), or inner product.
Support hybrid search: combine vector similarity with boolean metadata filtering and keyword BM25 search.

Non-Functional / SLA

Sub-15ms search latency across 100 Million high-dimensional vectors.
High recall accuracy (>95% top-K overlap compared to exact brute-force k-NN).
Real-time index freshness: newly upserted vectors must be searchable within 1 second.

2. Scale & Resource Estimates

Throughput: 10,000 vector search queries/sec. 2,000 vector upserts/sec.

Storage & RAM: 100 Million vectors * 1536 dimensions * 4 bytes (Float32) = 614 GB raw vectors. HNSW graph links add 1.5x overhead = ~1 Terabyte RAM.

3. System API Endpoints

POST /v1/vectors/upsert
Content-Type: application/json
{
  "collection": "technical_docs",
  "points": [
    {
      "id": "vec-881a",
      "vector": [0.024, -0.091, 0.412, ...], // 1536 float values
      "payload": { "doc_id": "doc-99", "category": "architecture", "access_level": 3 }
    }
  ]
}

POST /v1/vectors/search
Content-Type: application/json
{
  "collection": "technical_docs",
  "vector": [0.021, -0.088, 0.405, ...],
  "limit": 10,
  "filter": { "category": { "eq": "architecture" } }
}
Response:
{
  "results": [
    { "id": "vec-881a", "score": 0.964, "payload": { "doc_id": "doc-99" } }
  ]
}

4. Database Schema Design

Table: vector_collections (Metadata Store)
- collection_name (VARCHAR - Primary Key)
- dimension (INTEGER) -- e.g. 1536
- distance_metric (ENUM: COSINE, EUCLIDEAN, DOT)
- index_type (VARCHAR) -- "HNSW"

In-Memory Segment Structure:
- HNSW Graph Layers: Layer 0 (dense base) to Layer 3 (sparse entry)
- Quantized Vectors: PQ (Product Quantization) codebook (M=96 sub-vectors)
- Payload Filter Index: RocksDB inverted list & Roaring Bitmaps

5. Interactive High-Level Architecture

graph TD Client["AI Agent / RAG Ingestion Pipeline"] --> Gateway["Vector DB Gateway"] Gateway --> WAL[("Write-Ahead Log (WAL)")] WAL --> MemTable["In-Memory Mutable Segment Buffer"] MemTable --> SegmentFlusher["Segment Flusher & Merger"] SegmentFlusher --> HNSW["Hierarchical Navigable Small World (HNSW) Graph"] SegmentFlusher --> PQ["Product Quantization (PQ) Compression"] Client -->|Nearest Neighbor Search Request| QueryCoord["Distributed Query Coordinator"] QueryCoord --> FilterEngine["Metadata Filter Engine: Bitmaps"] QueryCoord --> HNSW HNSW --> PQCompute["SIMD AVX-512 Distance Calculator"] PQCompute --> RankedTopK["Ranked Top-K Search Results"]

6. Scalability Deep Dive & Bottlenecks

Brute-force k-NN search requires computing distance against every vector in the dataset—an O(N * D) operation that takes seconds per query at scale. Modern vector engines (Qdrant, Milvus, Pinecone) use HNSW (Hierarchical Navigable Small World) graphs. HNSW constructs a multi-layer graph where top layers have long-range highway edges (similar to skip lists) and bottom layers contain dense local clusters. A query drops into the top layer, greedily traverses toward the nearest neighbor, and descends layer by layer to achieve logarithmic O(log N) search time in under 10ms.

7. Production Scaling Strategies

Product Quantization (PQ) Compression: Uncompressed 1536-dim Float32 vectors consume 6 KB each. Product Quantization divides vectors into 96 sub-spaces and replaces them with 1-byte centroid indexes, shrinking memory footprint by 95% from 6 KB to 96 bytes.

SIMD Hardware Vectorization: Execute cosine and dot product distance calculations using CPU AVX-512 or ARM Neon vector instructions. SIMD processes 16 vector dimensions per CPU cycle, accelerating distance comparisons by 10x.

LSM-Tree Immutable Segment Architecture: Incoming upserts write first to an append-only WAL and a mutable in-memory segment. Once full, segments are sealed and frozen into immutable HNSW indices, eliminating lock contention during search queries.

Single-Stage Filtered Search (ACORN): Rather than naive post-filtering (which searches top 100 vectors then filters, potentially returning zero matches), evaluate metadata filters dynamically during graph traversal using Roaring Bitmaps.

Consistent Hashing Across Shards: Shard vector collections across nodes based on vector ID or partition keys. A Query Coordinator broadcasts searches to relevant shards in parallel and merges top-K candidate lists via a priority queue.

Disk-Backed Storage with DiskANN: For massive collections where RAM cost is prohibitive, store the full vector graph on fast NVMe SSDs and keep only compressed PQ vectors in RAM (DiskANN algorithm), achieving sub-20ms search on petabyte datasets.

8. Runnable Production Starter Code

Scaffold

System Design Architectures