Tarpitting | ScrapeRouter

Tarpitting is a defensive trick where a server deliberately slows down or traps a client instead of blocking it cleanly. The point is to waste the scraper, spammer, or scanner’s time and resources, which matters because a slow failure can be more expensive than a fast one in production.

Examples

A simple version is a site that accepts a connection, then responds painfully slowly so a bot ties up sockets, threads, or proxy bandwidth.

A more aggressive version is an AI-scraper tarpit that keeps returning endless low-value pages or fake links so the crawler burns requests without getting useful data.

curl -N https://example.com/trap

In scraping, the annoying part is that this may not look like a hard block at first. You just see requests hanging, timeouts creeping up, and throughput falling off a cliff.

Practical tips

Watch for slow success and slow failure patterns: long TTFB, connections that stay open too long, repeated pagination with little new data.
Treat tarpitting differently from normal rate limiting: a 429 is honest, a tarpit burns more CPU, proxy spend, and wall-clock time.
Set hard client timeouts so bad targets cannot pin your workers forever.
Track per-domain latency distributions: if one target suddenly shifts from 2 seconds to 45 seconds, something changed.
Rotate strategy, not just IPs: different headers, browser mode, request pacing, or a different fetch path may matter more than raw proxy churn.
If you are scraping at scale, isolate stuck jobs so one tarpit-heavy domain does not back up the whole queue.

import requests

try:
    r = requests.get("https://example.com", timeout=(10, 20))
    print(r.status_code)
except requests.Timeout:
    print("Possible tarpit or severely degraded target")

With ScrapeRouter, this is the kind of thing you want the router layer helping with: detect bad fetch paths early, shift strategy, and stop wasting expensive retries on requests that are technically alive but operationally dead.

Use cases

Anti-spam systems: SMTP services historically used tarpits to slow bulk senders instead of rejecting them immediately.
Bot and scraper defense: sites delay responses, throttle pages into near-useless speed, or feed crawlers junk paths.
AI crawler resistance: some sites now build infinite or nonsense link graphs specifically to waste crawler resources.
Security monitoring: defenders sometimes use tarpit-style behavior to slow scanners and make abuse less efficient.
Scraping operations: teams need to distinguish tarpitting from ordinary slowness, otherwise they keep retrying and paying for nothing.