API

Examples

• Calling a weather API to get the current temperature for a city
• Sending a payment request to Stripe's API from your checkout flow
• Using a scraping API to fetch a page without managing proxies or browser infrastructure yourself

import requests

response = requests.get(
    "https://api.example.com/products",
    headers={"Authorization": "Bearer token123"}
)

print(response.status_code)
print(response.json())

curl "https://api.example.com/products" \
  -H "Authorization: Bearer token123"

Practical tips

• Read the request format, authentication, rate limits, and response schema before you integrate. Most API problems are not mysterious. They are usually one of those four things.
• Treat the response format as a contract, but assume versions can change over time.
• Handle failures explicitly:
• timeouts
• retries
• 4xx errors
• 5xx errors
• rate limiting
• Log both the request metadata and the response status when debugging. Otherwise you end up guessing.
• If you depend on multiple providers, normalize responses behind your own interface or use a shared schema layer so downstream code does not have to change every time you switch.

import requests

try:
    response = requests.get("https://api.example.com/products", timeout=10)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.Timeout:
    print("Request timed out")
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e}")

Use cases

• Internal services talking to each other in a backend system
• A frontend app requesting data from a backend
• Sending messages through email, payments, or SMS providers
• Pulling data from SaaS tools like GitHub, Stripe, or Shopify
• Fetching web pages or structured data through a scraping API instead of managing raw scraping infrastructure yourself

Where it matters

Web scraping APIs matter because scraping is not just about making a request. In production, you run into blocking, rate limits, JavaScript rendering, proxy selection, retries, and provider-specific response formats.

What people do today is wire directly into one provider, get it working once, and then keep patching around edge cases. This works until the target changes, pricing changes, or the provider stops being a good fit. Then you have to rewrite both the fetch layer and parts of the downstream code.

The point is not just having an API. The point is having a stable interface between your application and the ugly part of scraping. That is why shared schemas and routing layers matter. They keep route changes out of application code and reduce lock-in.

If your scraping needs are simple and a plain request works, you probably do not need much here. But once you are scraping at scale, the API boundary becomes operationally important because it affects maintenance, debugging, cost, and how painful provider changes become.