Design Twitter / X
High-fanout social media feed with timeline generation, caching, and trending tag filters.
1. System Requirements
- Users post tweets (text/images/video).
- Home timeline displaying tweets of followed accounts.
- Follow/unfollow other accounts.
- Display trending topics and search index.
- High availability for reading timelines (read-heavy system).
- Low latency post execution (< 200ms tweet posting).
- Scalable fanout processing for celebrity accounts.
2. Scale & Resource Estimates
Throughput: 300 Million DAU. 500 Million tweets/day = 5.8k writes/sec. Average timeline reads = 100,000 requests/sec.
Storage & RAM: 500M tweets * 200 bytes text = 100 GB/day. Video/image storage is hosted separately on CDN caches.
3. System API Endpoints
// 1. Post a new tweet
POST /api/v1/tweets
Request: { user_id: "usr-4412", content: "Building the future with AI agents!" }
Response: { tweet_id: "twt-89324" }
// 2. Fetch timeline
GET /api/v1/timeline?user_id=usr-4412
Response: { tweets: [ { tweet_id: "twt-89324", user_id: "usr-4412", content: "..." } ] }4. Database Schema Design
Table: tweets (PostgreSQL / sharded by tweet_id)
- tweet_id (UUID - Primary Key)
- user_id (UUID - Index)
- content (VARCHAR(280))
- created_at (TIMESTAMP)
Table: timelines (Redis Lists - Memory Cache)
- Key: user_timeline:{user_id}
- Value: List of [tweet_id1, tweet_id2, ...]5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Features a **Fan-out Write Pipeline**. When a tweet is posted, it is queued in Kafka. The Fan-out Service fetches the user's follower list and pushes the tweet ID into every follower's active timeline in Redis (Push model). To handle popular accounts (e.g. 50M followers), the Fan-out Service skips pushing and instead stores the celebrity's tweet IDs separately; the read path then dynamically pulls and merges celebrity tweets with the user's timeline (Pull model).
7. Production Scaling Strategies
Fan-out Worker Pool: Fan-out service runs 500+ workers consuming from Kafka. Each worker handles one tweet → pushes to follower timelines in Redis. For a user with 1M followers, fan-out completes in ~2 seconds via parallel Redis LPUSH pipelines.
Timeline Cache: Each user's home timeline is a Redis list of the most recent 800 tweet IDs. On read, fetch tweet objects from a separate tweet cache (Memcached). Total Redis footprint: 500M users × 800 IDs × 8 bytes ≈ 3.2 TB — sharded across 200 Redis nodes.
Celebrity Threshold: Accounts with >500K followers are marked as 'celebrity'. Their tweets bypass fan-out entirely. At read-time, the Timeline Service merges the user's push-timeline with pulled celebrity tweets, sorted by relevance score (engagement + recency).
Search Indexing: Tweets are indexed in real-time into Elasticsearch via Kafka. Sharded by time (daily indices) with 30-day retention. Older tweets archived to cold storage with reduced search capability. Handles 500K search queries/sec.
Trending Topics: A Flink streaming job aggregates hashtag counts over 5-minute tumbling windows. Trending scores use a decay function (recent mentions weighted higher). Top 50 trends per-region cached in Redis, refreshed every 30 seconds.
Rate Limiting: Token bucket per-user: 300 tweets/3hr, 1000 API calls/15min. Implemented via Redis INCR with TTL. Prevents spam bots and protects fan-out workers from write amplification attacks.
8. Runnable Production Starter Code
Scaffold