Design Instagram
Social media feed generation service with photo uploads and content delivery network caching.
1. System Requirements
- Users upload high-resolution photos with captions.
- Display home feeds showing photos from followed accounts.
- Follow other users and search posts.
- Like and comment on posts.
- High availability for feed viewing.
- Low-latency feed load times (< 500ms).
- High durability of uploaded photos (no loss).
2. Scale & Resource Estimates
Throughput: 500 Million Daily Active Users (DAU). 50 Million photo uploads/day = ~580 uploads/sec. Average feed reads = 10,000 requests/sec.
Storage & RAM: 50M uploads/day * 2MB average size = 100 Terabytes/day. CDN origin shielded object storage.
3. System API Endpoints
// 1. Upload media item
POST /api/v1/posts/upload
Request: Multipart Form (image binary + caption)
Response: { post_id: "pst-1102", media_url: "https://media.aiagent.org/img-1102.jpg" }
// 2. Retrieve Home Feed
GET /api/v1/feed?page=1&limit=20
Response: { posts: [ { post_id: "pst-1102", author: "usr-441", media_url: "...", likes: 250 } ] }4. Database Schema Design
Table: posts (PostgreSQL - sharded by user_id)
- post_id (UUID - Primary Key)
- user_id (UUID - Sharding Key)
- media_url (VARCHAR)
- caption (TEXT)
- created_at (TIMESTAMP)
Table: follows (PostgreSQL - sharded by follower_id)
- follower_id (UUID - Primary Key)
- followee_id (UUID - Index)5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Uses a Hybrid Pull/Push Feed Generation model. Normal user feeds are pre-computed in Redis (Push model) when a user uploads a post. For popular celebrity accounts (e.g. 1M+ followers), their posts are not pushed to followers' feeds; instead, the Feed Service pulls their posts dynamically at read-time (Pull model) and merges them with the user's pre-computed feed, preventing write amplification.
7. Production Scaling Strategies
Feed Fan-out Threshold: Push model for users with <10K followers (write to each follower's Redis feed list). Pull model for celebrities >10K followers. At read time, merge the user's push-feed with pulled celebrity posts — sorted by timestamp + engagement score.
Image Pipeline: Uploads go to S3, triggering a Lambda/worker that generates 6 resolutions (thumbnail 150px → full 1080px). Store all variants on CDN. Serve via responsive <img srcset> — mobile gets 640px, desktop gets 1080px. Saves 60% bandwidth.
CDN Edge Caching: Serve images from 200+ CDN PoPs worldwide. Cache-Control: immutable for media URLs (content-addressed). Feed API responses cached at edge for 30s with stale-while-revalidate. Reduces origin load by 95%.
Database Sharding: Shard posts table by user_id (hash mod 64 shards). Likes/comments sharded by post_id. Use Vitess or PlanetScale for transparent MySQL sharding. Cross-shard queries (explore page) served from Elasticsearch replicas.
Explore/Recommendation: Batch compute user embeddings via Spark nightly. Store in a vector index (FAISS). Explore page queries FAISS for similar content embeddings, ranked by engagement signals. Cached per-user for 1 hour.
Write Amplification Control: Fan-out workers process post events from Kafka. Rate-limit fan-out to 50K writes/sec per worker. For a 10M-follower account, skip push entirely — rely on pull. This bounds worst-case fan-out latency to <5 minutes for normal users.
8. Runnable Production Starter Code
Scaffold