Architecture Blueprint

Design Distributed Task Scheduler & Job Orchestrator

Fault-tolerant distributed job scheduling platform managing millions of timed tasks, cron workflows, and delayed executions using Hashed Hierarchical Timing Wheels and consensus leader election.

1. System Requirements

Functional

Schedule one-time delayed jobs (e.g. send email in 45 minutes) and recurring cron jobs (e.g. daily billing reconciliation at midnight).
Execute scheduled tasks with high precision within seconds of their target trigger timestamp.
Track task execution statuses (PENDING, RUNNING, COMPLETED, FAILED) with automatic retries and dead-letter queues.

Non-Functional / SLA

Zero lost jobs: guaranteed at-least-once execution with idempotency protection.
High scalability: handle 50,000+ task triggers per second without master bottleneck.
Resilient to worker crashes, network partitions, and coordinator failover.

2. Scale & Resource Estimates

Throughput: 500 Million registered scheduled tasks. 40,000 tasks firing per second during peak hours.

Storage & RAM: Task definitions & payloads: 500M * 1 KB = 500 GB persistent database storage in PostgreSQL / CockroachDB.

3. System API Endpoints

POST /v1/jobs/schedule
Request:
{
  "target_endpoint": "https://api.internal/orders/cancel-expired",
  "payload": { "order_id": "ord-8812" },
  "trigger_type": "DELAYED", // DELAYED or CRON
  "execute_at": "2026-10-22T14:30:00Z",
  "max_retries": 3,
  "idempotency_key": "sch-idem-4412"
}
Response (201 Created):
{
  "job_id": "job-773129",
  "status": "SCHEDULED"
}

POST /v1/jobs/{job_id}/cancel
Response: { "status": "CANCELLED" }

4. Database Schema Design

Table: scheduled_jobs (PostgreSQL / Spanner)
- job_id (VARCHAR(64) - Primary Key)
- target_endpoint (VARCHAR(512))
- payload (JSONB)
- execute_at (TIMESTAMP WITH TIME ZONE - Indexed)
- cron_expression (VARCHAR(32))
- status (ENUM: PENDING, DISPATCHED, COMPLETED, FAILED, CANCELLED)
- retry_count (INTEGER)
- lease_owner (VARCHAR(64))
- lease_expires_at (TIMESTAMP)

Index:
CREATE INDEX idx_jobs_execute_status ON scheduled_jobs (execute_at, status) WHERE status = 'PENDING';

5. Interactive High-Level Architecture

graph TD ClientApp["External Microservices"] --> API["Scheduler API Gateway"] API --> JobDB[("Persistent Job Store: PostgreSQL / Spanner")] LeaderNode["Scheduler Master Coordinator: Raft Leader"] -->|Fetch 1-Hour Time Slice| JobDB LeaderNode --> TimeWheel["Hashed Hierarchical Timing Wheel: RAM"] TimeWheel -->|Tick Expiration: O(1)| DispatchQueue[("Distributed Queue: Kafka / SQS")] DispatchQueue --> WorkerA["Distributed Task Worker A"] DispatchQueue --> WorkerB["Distributed Task Worker B"] WorkerA --> Idempotency[("Redis Idempotency Check")] WorkerA --> ExecTool["Execute Task Target / Webhook"] WorkerA --> Heartbeat[("Heartbeat Tracker & Lease Renew")]

6. Scalability Deep Dive & Bottlenecks

Naive task schedulers query relational databases with polling loops (`SELECT * FROM jobs WHERE execute_at <= NOW() AND status = 'PENDING' FOR UPDATE SKIP LOCKED`). At 50,000 tasks/second, this crushes database disk I/O and creates severe lock contention. Modern systems decouple scheduling from storage using Hierarchical Hashed Timing Wheels (the algorithm powering Kafka timers and Linux kernels). A master coordinator pulls jobs in 1-hour time slices into tiered circular memory wheels (seconds, minutes, hours), achieving O(1) job insertion and expiration.

7. Production Scaling Strategies

Hierarchical Timing Wheel in Memory: Circular array data structure where an index pointer advances every second. Inserting and expiring tasks takes O(1) time without sorting or heap traversal, handling millions of active timers in memory.

Time-Slice Partitioning: Partition persistent storage by `execute_at` time buckets (e.g. daily tables). Coordinators scan only the active 1-hour time slice, preventing table scans over historical completed jobs.

Raft Consensus Leader Election: Coordinator nodes form a Raft cluster (using Etcd/Consul). Only the active leader manages the timing wheel; if the leader crashes, followers detect missing heartbeats and promote a new leader in <2 seconds.

Worker Lease Stealing: When workers pick up tasks from the queue, they acquire a 60-second lease in Redis. If a worker crashes mid-task, the lease expires and another worker automatically steals and retries the job.

Idempotent Target Execution: Because distributed networks guarantee at-least-once delivery, every scheduled task includes an `idempotency_key`. Target webhooks verify the key before processing to prevent duplicate side effects.

Adaptive Backpressure Queue Buffering: If worker fleets experience downstream saturation, dispatch queues dynamically throttle job ejection from the timing wheel, avoiding cascading failures.

8. Runnable Production Starter Code

Scaffold

System Design Architectures