How to Crawl at Least One Million URLs While Handling Memory Efficiently

[niteshver]

Discuss how to crawl 1M+ URLs efficiently while keeping memory usage low, focusing on URL queues, deduplication, concurrency, storage, and memory management.

[SohamKukreti]

Hi @niteshver at a scale of 1M+ URLs, there are a few things you should keep in mind:

1. Keep the URL queue on disk, not in python. Use something like SQLite with url as the primary key and a status column. Fetch maybe like 1000 at a time and mark the rows as results come in.
2. Crawl in chunks with stream=True. Use arun_many and write the results to disk as soon as they get streamed in.
3. Use MemoryAdaptiveDispatcher explicitly.

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=75.0,
    recovery_threshold_percent=65.0,
    max_session_permit=20,          # concurrent pages, tune per machine
    rate_limiter=RateLimiter(base_delay=(0.5, 1.5), max_retries=3),
)

It pauses new tasks when system memory crosses the threshold and resumes below recovery. The rate limiter is per domain and backs off on 429/503.

4. Make the browser cheap.

BrowserConfig(
    headless=True,
    text_mode=True,               # no images/fonts/JS; drop if you need JS
    light_mode=True,
    memory_saving_mode=True,      # cache discard + 512 MB V8 heap cap
    max_pages_before_recycle=500, # restart Chromium every N pages to shed leaks
)

If pages are server-rendered, skip Chromium entirely: AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()). Do an HTTP pass first, then re-queue only the empty results for a browser pass.

5. Turn off what you don't need.
   cache_mode=CacheMode.BYPASS (the built-in cache writes HTML + markdown for every URL to ~/.crawl4ai), screenshot=False, pdf=False, capture_network_requests=False, verbose=False.

6. Store only what you need.
   Write newline-delimited JSON (or Parquet) in rotating gzip files, one file per ~10k results. Keep markdown, not raw HTML, unless you truly need it.