Design Dropbox
Consistent distributed file synchronization storage system with chunked uploads and versioning.
1. System Requirements
- Upload and download files from any device.
- Automatic file synchronization across multiple active clients.
- Support offline editing and conflict resolution.
- Track file version histories.
- 100% data durability (strict ACID transaction updates).
- Strong consistency for concurrent edits.
- High bandwidth optimization (minimize redundant transfers).
2. Scale & Resource Estimates
Throughput: 100 Million Active Users. 1 Billion files uploaded/day. Average file size = 500 KB.
Storage & RAM: 1B uploads/day * 500 KB = 500 Terabytes/day raw. Reduced via chunking deduplication.
3. System API Endpoints
// 1. Request block upload token
POST /api/v1/files/upload/initiate
Request: { file_hash: "sha256-abc...", total_size: 16777216 }
Response: { upload_session_id: "sess-992", required_blocks: [ "blk-1", "blk-2" ] }
// 2. Upload block chunk
PUT /api/v1/files/upload/block?session_id=sess-992&block_id=blk-1
Request: Raw binary chunk (up to 4MB)
Response: { status: "chunk_saved" }4. Database Schema Design
Table: file_metadata (MySQL - sharded by namespace_id)
- file_id (UUID - Primary Key)
- name (VARCHAR)
- namespace_id (UUID)
- version (INTEGER)
- is_directory (BOOLEAN)
Table: file_blocks (MySQL)
- block_id (SHA-256 Hash - Primary Key)
- file_id (UUID - Foreign Key)
- block_index (INTEGER)
- size (INTEGER)5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Files are split into 4MB chunks and hashed using SHA-256. If a file is modified, only changed blocks are uploaded, which saves massive bandwidth. A local client Sync Engine monitors local file events. The Metadata Service routes updates to a transactional database (MySQL sharded). The Notification Service uses persistent HTTP Server-Sent Events (SSE) to push update signals to offline or active peer devices.
7. Production Scaling Strategies
Chunk Deduplication: SHA-256 hash each 4MB block. If the hash already exists in the block store, skip upload (reference existing block). At enterprise scale, dedup saves 40-60% storage. Cross-user dedup for shared files within an organization.
Metadata Sharding: Shard the file metadata MySQL database by user_id. Each shard tracks file tree, versions, and sharing ACLs. At 500M users, 128 shards each handle ~4M users. Vitess manages shard routing transparently.
Sync Conflict Resolution: When two clients edit the same file offline, the server detects conflicting versions via vector clocks. The later upload is saved as a 'conflicted copy' alongside the original. Users manually resolve — no silent data loss.
Notification Fan-out: When a file changes, notify all devices linked to that account via SSE (Server-Sent Events). For shared folders, fan-out to all collaborators' devices via Kafka topic per-folder. Batch notifications with 500ms debounce to avoid storms during bulk edits.
Block Storage Tiering: Hot blocks (accessed in last 30 days) on SSD-backed S3. Cold blocks moved to S3 Glacier after 90 days. Retrieval from cold storage takes 3-5 minutes — acceptable for rarely accessed files. Reduces storage costs by 70%.
Upload Throughput: Parallel chunk uploads (up to 8 concurrent) with resumable upload sessions. Each chunk upload is idempotent (retry-safe via chunk hash). At 10× scale, the upload service handles 500K chunks/sec across 200 upload workers.
8. Runnable Production Starter Code
Scaffold