Design TinyURL / Bitly
High-throughput URL shortener with persistent custom aliases and redirections.
1. System Requirements
- Generate a short, unique alias for a long URL.
- Redirect short aliases to their target long URLs via HTTP 301 Redirect.
- Support user-configured custom aliases.
- Redirect resolution must be sub-10ms.
- High write scalability, and strictly unique short codes.
2. Scale & Resource Estimates
Throughput: 100 Million short URLs generated/month. 10 Billion redirections/month = ~3,800 redirect reads/sec and ~38 writes/sec.
Storage & RAM: 100M URLs * 500 bytes = 50 GB storage/month. DB sharded by hash of the short code.
3. System API Endpoints
POST /api/v1/urls/shorten
Request: { long_url: "https://aiagent.org/designs/index.html", custom_alias: "sys-design" }
Response: { short_url: "https://bit.ly/sys-design" }
4. Database Schema Design
Table: url_mappings (MySQL - sharded by short_code hash)
- short_code (VARCHAR(8) - Primary Key)
- long_url (VARCHAR(2048))
- created_at (TIMESTAMP)
5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Utilizes a Key Generation Service (KGS) to guarantee unique short codes. KGS pre-allocates blocks of unique auto-incrementing numbers (mapped to Base62 short codes like 'aB9x2Z') to worker nodes, avoiding database sequence lock bottlenecks. Mappings are cached in Redis with a write-through strategy to ensure redirection hits resolve in under 10ms.
7. Production Scaling Strategies
KGS Pre-allocation: KGS generates keys in batches of 10K and assigns them to worker nodes. Each worker holds a local pool. When the pool drops below 1K, it requests a new batch. This eliminates database sequence contention entirely. At 100K URLs/sec, 10 KGS nodes handle the load.
Redis Cache-Aside: On redirect, check Redis first (cache hit ≈ 99.5% for popular URLs). Cache miss → fetch from PostgreSQL, write to Redis with 30-day TTL. Hot URLs (>100 clicks/day) are pinned in cache permanently. Total Redis footprint: ~500M entries × 200 bytes ≈ 100 GB.
301 vs 302 Redirect Strategy: Use 301 (permanent) for SEO-critical URLs — browsers cache the redirect. Use 302 (temporary) for analytics-tracked URLs — every click hits the server for tracking. Default to 302 to ensure click analytics accuracy.
Analytics Pipeline: Every click event is published to Kafka. A Flink streaming job aggregates clicks by short_code, referrer, geo, device. Stored in ClickHouse (columnar DB) for real-time analytics dashboards. Supports 1M+ clicks/sec ingestion.
Database Sharding: Shard the URL mapping table by short_code hash (mod 32). Each shard holds ~150M mappings. Read-heavy workload (1000:1 read:write ratio) — add read replicas per shard. PostgreSQL with connection pooling via PgBouncer.
Abuse Prevention: Rate limit URL creation to 100/hour per IP. Scan destination URLs against a malware/phishing blocklist (Google Safe Browsing API). Flag suspicious URLs for manual review. Auto-disable URLs with >50% bounce rate from security scanners.
8. Runnable Production Starter Code
Scaffold