Errors

An error is an HTTP status with a JSON body. Every error body has detail, a sentence you can show or log:

{"detail":"Invalid or expired API key"}

A request whose input is wrong (400) also has message (the same sentence) and errors, which says which field was refused:

{"detail":"limit: Enter 1000000 or less.","message":"limit: Enter 1000000 or less.","errors":{"config":["limit: Enter 1000000 or less."]}}

A few older answers use error instead of detail, for example a job that can’t be retried. Read detail, then message, then error; the Python SDK does. The examples on this page are real answers from the API.

Status codes

Status	Means	What to do
`400`	The request or the job’s input is wrong.	Fix the field `errors` names. Don’t retry unchanged.
`401`	No API key, or the key is wrong, revoked or expired.	Check the `X-API-Key` header. See Authentication.
`402`	The job’s estimate exceeds your balance, or pending and running jobs leave too much credit reserved.	Add credits, ask for less, or wait for jobs to finish. See below.
`403`	The key lacks the scope this needs.	Use a key with the scope `detail` names.
`404`	Nothing with that id in your organization.	Check the id, and that the key belongs to the right organization.
`409`	The object is still in use, for example an environment that still has keys.	Remove what refers to it first.
`422`	Every target in the job is on our exclusion list: people who asked not to be collected.	Remove those targets. The body lists them in `excluded_targets`.
`429`	Too many requests for this key.	Wait for `Retry-After` seconds. See below.
`500`	Something failed on our side.	Retry with backoff. If it persists, write to support with the time and endpoint.

A job that fails is not an HTTP error: creating it answered 201, and its outcome is in its status and error_code (Jobs and results). Two refusals carry amounts in detail: a 402 names the cost needed and your balance (and, when pending or running jobs hold credit, the amount they reserve), and a 400 for a job over its maxTotalChargeUsd names the estimate, the cap and how many results the cap buys.

Examples

400: an input the scraper doesn't take

{"detail":"Config for instagram scraper must include at least one of: directUrls, usernames, username, hashtags. Hint: \"profiles\" is not a valid input for instagram -- use usernames instead.","message":"Config for instagram scraper must include at least one of: directUrls, usernames, username, hashtags. Hint: \"profiles\" is not a valid input for instagram -- use usernames instead.","errors":{"config":["Config for instagram scraper must include at least one of: directUrls, usernames, username, hashtags. Hint: \"profiles\" is not a valid input for instagram -- use usernames instead."]}}

401: no key

{"detail":"Authentication credentials were not provided."}

403: missing scope

{"detail":"Insufficient scope. Required: admin"}

404: unknown id

{"detail":"No ScraperJob matches the given query."}

429: rate limited

HTTP/1.1 429 Too Many Requests
Retry-After: 3652
X-RateLimit-Limit: 1
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 3651

{"detail":"Request was throttled. Expected available in 3652 seconds."}

Taken from a key whose limit was set to 1 request an hour.

402: not enough credit

When a job is created, its estimated cost is checked against your organization’s balance. The API refuses it with 402 Payment Required if the balance can’t cover the estimate, or if the quoted estimated_cost of pending and running jobs plus its own exceeds 10 times the balance. The job isn’t created: nothing is charged and nothing runs. The body has only detail, in these formats (the amounts are illustrative):

{"detail":"Insufficient credits. Need $4.80, have $2.00. Please top up your credits."}

{"detail":"Insufficient credits. Need $4.80, have $250.00, and $2,498.40 is already reserved by 520 pending or running jobs. Please top up your credits, or wait for those jobs to finish."}

The estimate is the scraper’s rate times the results the job’s limits allow for, counting every target, and at most 50 per target per day on a crawl with a recent date filter (Billing). A smaller limit, a shorter date window or a maxTotalChargeUsd lowers it. To handle it:

Back off for a minute or more if credit is reserved by pending or running jobs, and wait for some to finish.
Add credits (Billing), or lower the job’s limits if its own estimate exceeds the balance.
Retry the same request. No job was created, so the retry is safe.

POST /api/v1/jobs/{id}/retry/ checks credit the same way. To be warned before you run out, subscribe a webhook to account.low_balance.

import os

from gofetch import GoFetchClient, InsufficientCreditsError

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])

try:
    run = client.actor("instagram").start(run_input={"usernames": ["nasa"], "limit": 10})
    print("started", run["id"])
except InsufficientCreditsError as error:
    print("Out of credit:", error.message)

429: too many requests

Each API key has a request limit per hour (Rate limits). Past it, the API answers 429 with Retry-After: the seconds until a request is allowed again. Wait that long, then retry. The Python SDK does this by itself, up to 3 times.

import os
import time
import urllib.request
from urllib.error import HTTPError

request = urllib.request.Request(
    "https://go-fetch.io/api/v1/jobs/?limit=1",
    headers={"X-API-Key": os.environ["GOFETCH_API_KEY"]},
)
for attempt in range(3):
    try:
        with urllib.request.urlopen(request) as response:
            remaining = response.headers["X-RateLimit-Remaining"]
            print(response.status, remaining, "requests left this hour")
            break
    except HTTPError as error:
        if error.code != 429:
            raise
        time.sleep(int(error.headers.get("Retry-After", "60")))

Retrying safely

Retry 429, 500, 502, 503, 504 and network errors with backoff. Don’t retry other 4xx unchanged.
Creating a job is not idempotent: if a create request timed out, list your recent jobs (GET /api/v1/jobs/?limit=5) before you send it again, or you may run the job twice.

Get started

Concepts

Scrapers

Python SDK

Status codes

Examples

402: not enough credit

429: too many requests

Retrying safely

​Status codes

​Examples

​402: not enough credit

​429: too many requests

​Retrying safely

Status codes

Examples

402: not enough credit

429: too many requests

Retrying safely