Parallel HTTP request dengan asyncio + httpx + Semaphore

Pernah perlu hit ribuan URL untuk audit atau bulk operation? Sequential bisa habiskan satu jam. Unbounded parallel bisa ngakitan target server. Bottleneck-controlled parallel sweet spot.

Kode

import asyncio
import httpx
from typing import List

async def fetch(client: httpx.AsyncClient, url: str, sem: asyncio.Semaphore) -> dict:
    """Fetch single URL with semaphore-controlled concurrency."""
    async with sem:
        try:
            r = await client.get(url, timeout=10.0)
            return {
                "url": url,
                "status": r.status_code,
                "size": len(r.content),
                "error": None,
            }
        except Exception as e:
            return {
                "url": url,
                "status": 0,
                "size": 0,
                "error": str(e),
            }

async def fetch_all(urls: List[str], concurrency: int = 10) -> List[dict]:
    """Fetch all URLs in parallel with max `concurrency` simultan."""
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(
        follow_redirects=True,
        headers={"User-Agent": "Mozilla/5.0 (Bot)"},
    ) as client:
        tasks = [fetch(client, url, sem) for url in urls]
        return await asyncio.gather(*tasks)

# Pemakaian
if __name__ == "__main__":
    urls = [
        f"https://example.com/page/{i}" for i in range(100)
    ]
    
    results = asyncio.run(fetch_all(urls, concurrency=5))
    
    success = sum(1 for r in results if r["error"] is None and r["status"] == 200)
    print(f"Success: {success}/{len(results)}")

Performance comparison

100 URL ke https://httpbin.org/delay/1 (1 detik delay per request):

Approach	Time
Sequential	102 detik
asyncio + concurrency=5	22 detik
asyncio + concurrency=10	11 detik
asyncio + concurrency=50	4 detik (tapi server complain)
asyncio + unbounded	3 detik (tapi server return 429)

Sweet spot: concurrency 5-10 untuk hampir semua target. Yang lebih: kontraproduktif (rate limit, server overload).

Kapan dipakai

• Bulk fetch dari internal API (sitemap scrape, dashboard data)
• Validate ribuan URL (audit broken links)
• Bulk RSS feed reader
• Background ETL job

Kapan JANGAN dipakai

• Hit external API yang strict rate limit (Twitter, Stripe): pakai library dengan built-in backoff (tenacity, aiolimiter)
• Single URL fetch: pakai requests biasa (asyncio overhead untuk satu request: wasteful)
• CPU-bound work: asyncio untuk I/O. Untuk CPU, pakai multiprocessing atau concurrent.futures.ProcessPoolExecutor

Catatan

• async with sem: yield control kalau slot habis. Penting — tanpa async with, semaphore tidak handle exception cleanup.
• follow_redirects=True default httpx — match requests default. Set False kalau Anda ingin track 301/302 manually.
• User-Agent: target server kadang block default httpx UA. Set yang manusiawi.
• Exception inside gather: dengan return_exceptions=False (default), satu exception abort semua. Pakai return_exceptions=True kalau ingin tetap collect partial results.

Variasi: progress bar

from tqdm.asyncio import tqdm_asyncio

async def fetch_all_with_progress(urls, concurrency=10):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient() as client:
        tasks = [fetch(client, url, sem) for url in urls]
        return await tqdm_asyncio.gather(*tasks, desc="Fetching")

tqdm_asyncio show progress bar yang update real-time. Berguna untuk jobs > 100 URL.

Untuk production scraping besar (10k+ URL/jam), pakai dedicated tool seperti Scrapy. Snippet ini untuk ad-hoc bulk operation.