Architecture Blueprint

Design Real-Time Observability & Telemetry Pipeline (Datadog)

Massive-scale distributed telemetry ingestion platform processing millions of logs, OpenTelemetry traces, and time-series metrics per second with ClickHouse columnar storage and real-time alert evaluation.

1. System Requirements

Functional

Ingest structured logs, distributed trace spans (OpenTelemetry OTLP), and high-frequency metrics across 50,000+ host agents.
Provide sub-second full-text log search and distributed trace waterfall visualizations.
Evaluate real-time alert rules (e.g., error rate > 5% over 1 minute) and trigger automated PagerDuty incidents.

Non-Functional / SLA

Sub-500ms P95 query response time across billions of historical log lines and trace spans.
High ingestion resilience: zero log drops during sudden network partition flushes (buffer via Kafka).
High compression ratio (>80% reduction) to minimize multi-petabyte disk storage costs.

2. Scale & Resource Estimates

Throughput: 50,000 servers. 5 Million log lines/sec, 2 Million trace spans/sec, 10 Million metrics/sec (~60 GB/sec ingestion stream).

Storage & RAM: Raw uncompressed telemetry: ~5 Terabytes/hour. With columnar ZSTD compression = ~750 Gigabytes/hour (18 TB/day).

3. System API Endpoints

POST /v1/telemetry/v1/logs
Content-Type: application/x-protobuf
Headers: { "X-API-Key": "otlp-key-8812a" }
[Binary Protobuf Stream of OTLP Log Records]

POST /api/v1/query/logs
Request:
{
  "query": "service:payment-gateway AND status:500 AND @latency_ms:>500",
  "start_time": "now-15m",
  "end_time": "now",
  "limit": 50
}
Response:
{
  "total_matches": 142,
  "events": [
    { "timestamp": "2026-10-22T14:12:01.812Z", "service": "payment-gateway", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "message": "Connection pool timeout" }
  ]
}

4. Database Schema Design

Table: otel_logs (ClickHouse MergeTree)
- timestamp (DateTime64(3, 'UTC') - Codec: DoubleDelta)
- service_name (LowCardinality(String))
- trace_id (FixedString(32))
- span_id (FixedString(16))
- severity (LowCardinality(String))
- body (String - Codec: ZSTD(3))
- attributes (Map(String, String))
ORDER BY (service_name, severity, toDate(timestamp), timestamp)
SETTINGS index_granularity = 8192;

Table: otel_traces (ClickHouse MergeTree)
- trace_id (FixedString(32))
- span_id (FixedString(16))
- parent_span_id (FixedString(16))
- service_name (LowCardinality(String))
- operation_name (LowCardinality(String))
- start_time (DateTime64(6, 'UTC'))
- duration_microseconds (UInt64)
- status_code (UInt8)
ORDER BY (service_name, operation_name, start_time);

5. Interactive High-Level Architecture

graph TD HostAgents["Host OpenTelemetry Collectors"] -->|OTLP Protocol: gRPC| Gateway["Telemetry Ingestion Gateway"] Gateway --> KafkaBuffer[("Kafka Ingestion Queue: Logs, Traces, Metrics")] KafkaBuffer --> StreamProcessor["Vector / Flink Stream Processors"] StreamProcessor --> AlertEngine["Real-time Alert Rule Evaluator"] AlertEngine -->|Threshold Breach| PagerDuty["PagerDuty / Slack Alerts"] StreamProcessor --> ClickHouse[("ClickHouse Columnar Storage: ZSTD")] ClickHouse --> RawLogs["otel_logs: Sparse Primary Index"] ClickHouse --> Traces["otel_traces: Span Tree Storage"] ClickHouse --> Metrics["otel_metrics: Gorilla Time-Series Rollup"] Dashboard["Grafana / Query UI"] --> ClickHouse

6. Scalability Deep Dive & Bottlenecks

Traditional document stores (like standard Elasticsearch clusters) fall over at multi-million event/second ingestion because inverted indexing on every JSON field consumes massive CPU and triples disk footprints. Modern telemetry stacks (Datadog, ClickHouse, VictoriaMetrics) leverage columnar storage with sparse primary indexes and specialized encoding codecs (Gorilla XOR for float metrics, DoubleDelta for timestamps, ZSTD for text). Columnar layouts allow analytical queries (`SELECT avg(latency) WHERE service = 'auth'`) to read only the relevant single column off disk, scanning 100M rows in under 200ms.

7. Production Scaling Strategies

ClickHouse Columnar Compression: Data is organized by column rather than row. Similar data types compress exceptionally well (up to 10:1 ratio with ZSTD and DoubleDelta), reducing 100 TB of raw logs to 10 TB on disk.

Tail-Based Trace Sampling: Tracing every request generates petabytes of redundant data. Trace collectors buffer all spans in memory until a trace completes: keep 100% of error traces (HTTP 5xx) and high-latency traces (P99 > 1s), but downsample successful 200 OK traces to 1%.

Gorilla Compression for Float Metrics: Numerical metrics use Facebook's Gorilla XOR compression algorithm, reducing 64-bit floating-point metrics to an average of 1.37 bytes per timestamp-value pair in RAM.

Streaming Alert Evaluation via Flink: Alert rules evaluate against incoming Kafka streams in memory using sliding windows. Incidents fire in under 2 seconds rather than relying on heavy scheduled database queries.

Vector Telemetry Agent Aggregation: Run lightweight Vector or FluentBit daemonsets on every host. Agents batch logs into 1 MB compressed gzip payloads before sending to origin gateways, slashing network overhead by 80%.

Tiered Storage Lifecycle: Hot logs reside on fast NVMe SSD ClickHouse nodes for 7 days. At day 8, background MergeTree processes move immutable data parts to S3/GCS object storage, dropping storage costs by 90% while retaining full queryability.

8. Runnable Production Starter Code

Scaffold

System Design Architectures