Architecture Blueprint

Design Web Crawler

High-throughput web scraping engine with URL prioritization queue, politeness filters, and page deduplication.

1. System Requirements

Functional

Ingest a seed list of URLs and crawl page contents.
Parse HTML to extract text content and nested links.
Avoid duplicate page crawls using URL deduplication.
Adhere to robots.txt permissions and domain crawl rates.

Non-Functional / SLA

Politeness policies (do not overwhelm target domains).
High scalability (process billions of pages per month).
Deduplication integrity (ignore near-identical page bodies).

2. Scale & Resource Estimates

Throughput: 10 Billion Web Pages crawled per month = ~3,800 pages per second. Average page size raw HTML = 100 KB.

Storage & RAM: 3,800 pages/sec * 100 KB = 380 Megabytes/second writes = 1 Petabyte/month total storage.

3. System API Endpoints

// Internal worker task queue payload
{
  "task_id": "crl-449301",
  "url": "https://aiagent.org/bootcamp",
  "depth": 3,
  "last_crawled_timestamp": 1783708872
}

4. Database Schema Design

Table: crawled_pages (HBase / Cassandra key-value store)
- url_hash (MD5 - Primary Key)
- raw_html (BLOB)
- crawled_at (TIMESTAMP)

Table: url_frontier (Redis Queue - Priority)
- Key: domain_queue:{domain}
- Value: Sorted Set of target URL links (Score: priority)

5. Interactive High-Level Architecture

6. Scalability Deep Dive & Bottlenecks

Uses a URL Frontier to organize pending links. To ensure politeness, the system routes URLs into dedicated queues based on domain name, with only one fetcher worker querying a single domain queue at a time (enforcing a delay between requests). To avoid duplicate crawling, a **Bloom Filter** checks if a URL has already been processed. A **SimHash algorithm** computes signatures of HTML content to identify and discard near-duplicate pages.

7. Production Scaling Strategies

URL Frontier Partitioning: Distribute the URL frontier across 100+ Redis nodes, partitioned by domain hash. Each partition has its own priority queue (Sorted Set). High-priority domains (news sites) get crawled more frequently. Frontier holds ~10B URLs at steady state.

Fetcher Worker Pool: Run 10K+ fetcher workers across multiple regions. Each worker handles one domain at a time with a politeness delay (1-5 seconds between requests). Workers run on spot instances — if terminated, the URL returns to the frontier automatically.

Deduplication at Scale: Use a distributed Bloom Filter (partitioned across 50 Redis nodes, 10B capacity, 0.1% false positive rate) to check if a URL was already crawled. Content-level dedup via SimHash — pages with >90% similarity are discarded.

DNS Resolution Caching: DNS lookups are a bottleneck at scale. Run a local DNS cache (each fetcher worker caches resolved IPs with 1-hour TTL). Reduces DNS queries from 1B/day to ~50M/day. Use DNS pre-fetching for domains in the frontier.

Content Processing Pipeline: Fetched HTML is pushed to Kafka. Downstream consumers: (1) link extractor → adds new URLs to frontier, (2) content indexer → builds inverted index, (3) PageRank compute → updates link graph. Each consumer scales independently.

Crawl Budget Management: Allocate crawl bandwidth per-domain based on site importance (PageRank) and freshness requirements. News sites get re-crawled every 15 minutes. Static sites get re-crawled weekly. robots.txt is respected and cached per-domain.

8. Runnable Production Starter Code

Scaffold

System Design Architectures