Elasticsearch & OpenSearch Tutorial — Search, Vector DB & Cluster Guide

1 Architecture & Apache Lucene Internals

📑

Inverted Index & Postings

Rather than scanning text document by document, Lucene maps every unique token to a sorted list of document IDs (postings list), term frequency (TF), and term positions. Enables sub-10ms lookups across billions of documents.

🧱

Immutable Segments

An index is composed of multiple immutable Lucene segments written to disk. Documents are never updated in-place; updates write a new document and flag the old ID in a bitset (.del file). Background threads periodically merge smaller segments into larger ones.

🔀

Shards & Write Path

An index is partitioned into Primary Shards and replicated into Replica Shards. Writes route via hash: shard = hash(_routing) % num_primaries. The primary writes to memory buffer & Translog (WAL) before replicating to replicas in parallel.

🖥️

Node Roles & Topologies

Production clusters separate workloads: Master-eligible (cluster state & quorum), Data nodes (hot/warm/cold shard storage & search), Coordinating nodes (scatter-gather request routing), and Ingest nodes (pre-processing pipelines).

📊

Doc Values & Columnar Store

While the inverted index optimizes search, sorting, aggregations, and script lookups require Doc Values. Doc Values store document-to-field values in an on-disk columnar format mapped into the OS filesystem page cache to avoid JVM heap pressure.

🔄

Near-Real-Time (NRT) Refresh

By default, Lucene opens newly written in-memory segments to search every 1s (the refresh interval). Commits (fsyncing Translog to disk) happen every 30 minutes or when Translog hits 512MB, decoupling search visibility from disk I/O cost.

🔍

Analyzers & Tokenizers

Text processing pipeline: Character Filters (strip HTML, replace regex), Tokenizer (split whitespace, words, n-grams), and Token Filters (lowercase, stopwords, stemming, synonyms). Custom analyzers power search precision.

🎯

BM25 Scoring Formula

Best Matching 25 is the default full-text relevance ranking. It calculates score using Term Frequency with saturation parameter k1=1.2 and Inverse Document Frequency normalized against document length with parameter b=0.75.

2 Elasticsearch vs. OpenSearch: In-Depth Comparison

The 2021 License Fork & Beyond

History & Context: Why the Fork Happened

In early 2021, Elastic NV relicensed Elasticsearch and Kibana from Apache 2.0 to the Server Side Public License (SSPL) and Elastic License v2 to prevent cloud providers (specifically AWS) from offering managed Elasticsearch without commercial contribution. In response, Amazon Web Services (AWS) along with a consortium of community contributors forked Elasticsearch 7.10.2 and Kibana 7.10.2 to establish OpenSearch and OpenSearch Dashboards under the permissive Apache 2.0 license. Today, both projects have diverged significantly with distinct feature sets, vector implementations, and cloud ecosystems.

Evaluation Dimension	Elasticsearch (Elastic NV)	OpenSearch (Linux Foundation / AWS)
License	Dual license: SSPL + Elastic License v2 (non-OSI)	100% Open Source under Apache 2.0
Vector Search Engine	Native `dense_vector` field with Lucene HNSW graph & Flat indexing	`knn_vector` plugin with choice of Faiss, NMSLIB, or Lucene backends
Proprietary AI Models	ELSER (Elastic Learned Sparse EncodeR) out-of-the-box	Neural Search & ML Commons connector (HuggingFace, Bedrock, SageMaker)
Hybrid Search Fusion	Native `rrf` (Reciprocal Rank Fusion) clause in search API	Search Pipeline with `normalization-processor` (Min-Max, L2, RRF)
Cloud Managed Services	Elastic Cloud (hosted on AWS, Azure, GCP)	Amazon OpenSearch Service & OpenSearch Serverless
Security & RBAC	Native X-Pack security (basic TLS/native auth included free)	OpenSearch Security Plugin (Apache 2.0, SAML, OIDC, fine-grained access)
Index Lifecycle	ILM (Index Lifecycle Management) + Data Streams	ISM (Index State Management) + Data Streams
Storage Tiering	Hot, Warm, Cold, Frozen (Searchable Snapshots)	Hot, Warm (UltraWarm backed by S3), Cold Storage (direct S3 queries)
Client SDKs	`elasticsearch` (Python, JS, Go, Java) with strict server check	`opensearch-py`, `opensearch-js`, Community SDKs
Best Suited For	Elastic ecosystem users, out-of-the-box ELSER, APM/Observability suite	AWS-native workloads, open-source compliance requirements, custom Faiss k-NN

3 Quickstart & Docker Compose Setups

⚡ Elasticsearch 8.x + Kibana

Dev Single-Node

Spin up an Elasticsearch 8 container with Kibana dashboard on localhost (security disabled for rapid local prototyping):

# docker-compose-elastic.yml
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
container_name: es-dev
environment:
- discovery.type=single-node
- xpack.security.enabled=false
- ES_JAVA_OPTS=-Xms2g -Xmx2g
ports:
- "9200:9200"
kibana:
image: docker.elastic.co/kibana/kibana:8.15.0
container_name: kibana-dev
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- "5601:5601"

API: http://localhost:9200 UI: http://localhost:5601

🔍 OpenSearch 2.x + Dashboards

Apache 2.0

Spin up an OpenSearch 2 node with k-NN plugin and OpenSearch Dashboards (security disabled for local dev):

# docker-compose-opensearch.yml
services:
opensearch:
image: opensearchproject/opensearch:2.16.0
container_name: opensearch-node
environment:
- discovery.type=single-node
- plugins.security.disabled=true
- OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g
ports:
- "9200:9200"
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.16.0
environment:
- OPENSEARCH_HOSTS=["http://opensearch:9200"]
ports:
- "5601:5601"

API: http://localhost:9200 UI: http://localhost:5601

Python Driver Installation Note:

Elasticsearch 8+ enforces official client headers, while OpenSearch requires its dedicated SDK.

pip install elasticsearch pip install opensearch-py

4 Hands-On Code Examples (Full-Text, Vectors & RAG)

1. Custom Mappings with Edge N-Gram Autocomplete & dense_vector

REST API / JSON DSL

# PUT /knowledge_base
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 2, "max_gram": 15
}
},
"analyzer": {
"agent_text_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "autocomplete_filter"]
}
}
}
},
"mappings": {
"properties": {
"title": { "type": "text", "analyzer": "agent_text_analyzer" },
"content": { "type": "text", "similarity": "BM25" },
"category": { "type": "keyword" },
"embedding": {
"type": "dense_vector",
"dims": 1536,
"index": true,
"similarity": "cosine",
"index_options": { "type": "hnsw", "m": 16, "ef_construction": 100 }
}
}
}
}

2. Production Python: Hybrid Search with Reciprocal Rank Fusion (RRF)

elasticsearch-py & OpenAI

# hybrid_rag_retriever.py
from elasticsearch import Elasticsearch
from openai import OpenAI
es = Elasticsearch("http://localhost:9200")
openai_client = OpenAI()
def hybrid_search(query_text: str, tenant_id: str, top_k: int = 5):
# 1. Generate query vector embedding
emb_resp = openai_client.embeddings.create(
model="text-embedding-3-small", input=query_text
)
query_vector = emb_resp.data[0].embedding
# 2. Execute Reciprocal Rank Fusion query
response = es.search(
index="knowledge_base",
retriever={
"rrf": {
"retrievers": [
# Branch A: Keyword BM25 Search with filter
{
"standard": {
"query": {
"bool": {
"must": [{"match": {"content": query_text}}],
"filter": [{"term": {"tenant_id": tenant_id}}]
}
}
}
},
# Branch B: Dense HNSW k-NN Vector Search
{
"knn": {
"field": "embedding",
"query_vector": query_vector,
"k": 20,
"num_candidates": 100,
"filter": {"term": {"tenant_id": tenant_id}}
}
}
],
"rank_constant": 60,
"rank_window_size": top_k
}
}
)
return [hit["_source"] for hit in response["hits"]["hits"]]

3. Exposing Search as an Autonomous Agent Tool

ReAct / Function Calling

Agents query search clusters dynamically. When structured filters (e.g. date ranges, categories, tenant isolation) are combined with natural language semantic recall, hallucinations are eliminated:

# agent_search_tool.py (JSON Schema for LLM Function Calling)
search_tool_schema = {
"type": "function",
"function": {
"name": "search_enterprise_knowledge",
"description": "Query company technical docs, runbooks, and bug tickets using hybrid semantic search.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Natural language query"},
"category": {"type": "string", "enum": ["infra", "api", "billing", "security"]},
"created_after": {"type": "string", "description": "ISO-8601 date cutoff (e.g. 2026-01-01)"}
},
"required": ["query"]
}
}
}

5 Interactive Sharding & Cluster Sizing Calculator

🧮 Production Capacity & Shard Planner

Calculate exact primary shards, replica allocations, storage footprint, and required JVM heap memory based on industry best practices (30GB–50GB shard limit, max 31GB heap with Compressed OOPs).

Daily Ingestion (GB / Day) Uncompressed raw logs or docs

Retention Period (Days) Active days kept in hot/warm tiers

Number of Replicas Shard copies for resilience & read scaling

Target Shard Size Lucence segment efficiency limit

Total Disk Required 7,200 GB +20% translog/merge buffer

Recommended Primary Shards 100 Across retention window

Total Active Shards 200 Primaries + Replicas

Recommended Data Nodes 4 Nodes ~31GB RAM / 1.8TB SSD each

6 Index Lifecycle Management (ILM / ISM)

1. Hot Tier NVMe SSD

Active index ingestion and recent query reads. High CPU and fast NVMe SSD storage. Rollover triggered after 50GB or 30 days.

2. Warm Tier Standard SSD

Read-only indices. Ingestion stops. Trigger _forcemerge to 1 segment per shard and shrink replica counts to reclaim cluster memory.

3. Cold Tier S3 / Object Store

Infrequently queried historical logs. Backed by Searchable Snapshots (Elasticsearch) or UltraWarm / Cold Storage in AWS OpenSearch, slashing storage costs by 80%.

4. Delete Phase Retention Cutoff

Automated deletion after compliance retention window (e.g., 90 or 365 days). Safely removes shards from cluster state without manual operator intervention.

7 Production Tuning & Performance Checklist

The 50% & 31GB JVM Heap Rule

Never allocate more than 50% of physical RAM to JVM heap. The other 50% must remain available for the OS kernel page cache to hold Lucene segment files. Always cap heap at < 31GB to maintain 32-bit Compressed Ordinary Object Pointers (Compressed OOPs).

Bulk Ingestion Tuning

During initial massive backfills: temporarily set "index.refresh_interval": "-1" and "index.number_of_replicas": 0. Batch requests in chunks of 5MB–15MB or 1,000–5,000 documents using the Bulk API.

Disk Watermark Safeguards

Elastic/OpenSearch halts operations when disks fill up: Low watermark (85%) prevents allocating new shards; High watermark (90%) moves existing shards; Flood stage (95%) locks indices into read-only mode (read_only_allow_delete).

Filter Context vs. Query Context

Use filter clauses instead of must when relevance score is not needed (e.g. status codes, tenant IDs, timestamps). Filters bypass Lucene BM25 scoring arithmetic and are automatically cached in node memory bitsets.

Avoid Deep Pagination (search_after)

Never page past 10,000 results using from: 10000, which triggers massive coordinating node heap exhaustion as every shard must sort from + size documents. Use search_after with a Point in Time (PIT) cursor for deep exports.

Fielddata Trap

Never enable fielddata: true on analyzed text fields to perform aggregations or sorting. Fielddata builds in-heap un-inverted structures that cause catastrophic JVM Out-Of-Memory (OOM) crashes. Use multi-field keyword sub-fields with Doc Values instead.

8 Frequently Asked Questions

Can I migrate from Elasticsearch 7.10 directly to OpenSearch? →

Yes. OpenSearch 1.x was forked directly from Elasticsearch 7.10.2. You can take an Elasticsearch 7.10 snapshot and restore it directly into an OpenSearch cluster, or point an OpenSearch cluster at compatible Lucene index files. For Elasticsearch 8.x, however, indices must be reindexed via Logstash or remote reindex API because Elasticsearch 8.x index formats contain proprietary changes incompatible with OpenSearch 2.x.

How does Elasticsearch's ELSER compare to OpenSearch's Neural Search? →

ELSER is Elastic's proprietary learned sparse retrieval model that expands queries into weighted vocabulary tokens without requiring dense vector indexes, working seamlessly within standard inverted index structures. Neural Search in OpenSearch enables uploading Hugging Face models (like BERT or Cohere) directly to the cluster via ML Commons or connecting to Amazon Bedrock, generating vector embeddings dynamically during document ingestion and search.

Why should I use Elasticsearch/OpenSearch for vectors instead of Pinecone or Qdrant? →

Standalone vector DBs excel at pure similarity search, but real-world AI applications demand Hybrid Search (combining exact keyword matching, SKU lookups, date filtering, security ACLs, and vector similarity). Elasticsearch and OpenSearch provide the industry's most battle-tested full-text, aggregations, and geo-filtering engines alongside native k-NN vector indexes, enabling unified queries without dual-system synchronization overhead.

What is the maximum number of shards a node can hold? →

A strict rule of thumb is no more than 20 to 25 shards per GB of JVM heap. A data node configured with a 30GB heap should hold a maximum of 600 to 750 shards. Exceeding this limit causes cluster state master updates to lag, slows down cluster recoveries, and degrades search latency due to coordinating node thread fan-out.

Why does Lucene require 50% of RAM for OS Page Cache? →

Lucene relies heavily on memory-mapped files (mmap) to access inverted index postings, term dictionaries, and Doc Values directly from the operating system kernel's file cache. If JVM heap is configured too large (e.g. 90% of RAM), the OS cannot cache Lucene segments in RAM, forcing the kernel to read segments from physical disk on every query, which degrades p99 latency by orders of magnitude.