SNI | ScrapeRouter

SNI stands for Server Name Indication, a TLS extension that lets a client say which hostname it wants before the HTTPS connection is fully set up. In practice, that matters because many sites share the same IP, and if the SNI value is wrong, missing, or blocked, the request can fail before scraping even gets to HTTP.

Examples

A browser or scraper connecting to https://example.com usually sends the hostname in the TLS handshake as SNI.

openssl s_client -connect example.com:443 -servername example.com

If you leave out the server name on a multi-tenant host, you can get the wrong certificate or a failed handshake.

openssl s_client -connect example.com:443

In Python, the hostname in the URL is typically used as SNI automatically by the TLS client.

import requests

r = requests.get("https://example.com", timeout=30)
print(r.status_code)

Practical tips

If an HTTPS target works in a browser but fails in your scraper, check the TLS handshake first: certificate mismatch, handshake reset, wrong hostname.
Don’t assume "port 443 is up" means the site is reachable: SNI can still be required for the connection to work.
When debugging, compare behavior with and without the -servername flag in openssl s_client.
In production scraping, weird intermittent failures sometimes come from proxy or client stacks sending the wrong hostname, or not sending SNI cleanly through the chain.
If you’re routing requests through scraping infrastructure, this is one of those details you usually don’t want to babysit yourself. ScrapeRouter handles the ugly transport-layer stuff so you’re not debugging TLS edge cases at 2 a.m.

# good: explicit SNI
openssl s_client -connect api.example.com:443 -servername api.example.com

# useful for checking certificate presented by the server
openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null

Use cases

Shared hosting and CDNs: multiple domains sit behind one IP, so the server needs SNI to know which certificate and site to serve.
Scraper debugging: a request fails before any HTTP response shows up, and the real issue is TLS negotiation, not your headers or parser.
Proxy chains: if a proxy, browser automation layer, or custom TLS client mishandles SNI, some targets will break in ways that look random until you inspect the handshake.
Network filtering: some firewalls and filtering systems inspect SNI because it is sent early in the connection setup.