Design YouTube
High-throughput video ingestion, adaptive bitrate transcoding, and CDN streaming distribution.
1. System Requirements
- Users upload raw video content.
- Stream video at multiple resolutions (1080p, 720p, 480p).
- Search videos and trace metrics (views, likes).
- Post comments on video playbacks.
- Extremely low playback buffering latency.
- High availability for streaming.
- Scalable video transcoding pipeline.
2. Scale & Resource Estimates
Throughput: 2 Billion Monthly Active Users. 500 hours of video uploaded every minute. Average video encoding size = 500 MB/hour.
Storage & RAM: 500 hours/min * 60 min * 500 MB = 15 Terabytes/hour raw. Multiplied across resolutions.
3. System API Endpoints
// 1. Initiate chunked video upload
POST /api/v1/video/upload/initiate
Request: { title: "Agentic AI tutorial", size: 1073741824 }
Response: { upload_token: "upl-abc...", chunk_url: "https://ingest.youtube.org/chunk" }
// 2. Stream segment info
GET /api/v1/video/manifest?video_id=vid-9932
Response: { formats: [ { resolution: "1080p", manifest_url: "https://cdn.youtube.org/vid-9932/1080p.m3u8" } ] }4. Database Schema Design
Table: video_metadata (MySQL - sharded by video_id)
- video_id (UUID - Primary Key)
- title (VARCHAR)
- uploader_id (UUID)
- description (TEXT)
- views_count (BIGINT)
- created_at (TIMESTAMP)
Table: video_chunks (Blob Storage paths)
- video_id (UUID)
- resolution (ENUM: 1080p, 720p)
- segment_index (INTEGER)
- file_path (VARCHAR)5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Uses an **Adaptive Bitrate Streaming (ABR)** pipeline. Raw files are uploaded to S3, triggering transcoding jobs via Kafka. The Transcoding Service splits the video into small chunk segments (e.g. 5-second pieces) and encodes them in multiple formats (H.264, VP9) at different resolutions. Clients stream chunks using protocols like HLS or MPEG-DASH, and the client media player dynamically adjusts resolution based on network bandwidth.
7. Production Scaling Strategies
Transcoding Farm: Each uploaded video spawns 20-30 transcoding jobs (6 resolutions × 3-5 codecs). Run on spot/preemptible GPU instances. A 10-minute 4K video takes ~15 minutes to transcode. At 500 hours uploaded/min, run 10K+ concurrent transcoding workers.
CDN Edge Replication: Pre-replicate popular videos (top 10% by views) to all CDN PoPs. Long-tail videos cached on-demand with LRU eviction. A video going viral triggers automated PoP pre-warming via predictive ML models (view velocity detection).
Recommendation Pre-compute: Nightly Spark batch job computes 'related videos' for the top 1M videos using collaborative filtering. Stored in Redis. For long-tail videos, compute recommendations on-the-fly using approximate nearest neighbor (ANN) on video embeddings.
Comment Sharding: Comments sharded by video_id in Cassandra. Top-level comments sorted by engagement score. Reply trees stored as nested partition keys. At 10M comments/day, each of 50 Cassandra nodes handles ~200K writes/day.
Live Streaming: Ingest RTMP streams, segment into 2-second HLS chunks in real-time via FFmpeg workers. Distribute chunks to edge CDN within 3 seconds of capture. Scale ingest servers per-region based on concurrent live streams (up to 100K simultaneous).
Storage Cost Management: Store original raw file + top 3 popular resolutions on hot storage. Transcode remaining resolutions on-demand for first request, then cache. Delete rarely-viewed renditions after 90 days. Saves 40% storage costs at exabyte scale.
8. Runnable Production Starter Code
Scaffold