Webhooks

A webhook is a URL of yours that we call when an event happens, so you don’t have to poll. Every call is signed with HMAC-SHA256.

Events

Event	Sent when
`job.completed`	A job finished. The body carries its final charge.
`job.failed`	A job failed and no automatic retry is coming.
`job.timed_out`	A job ran past its `timeout`.
`job.cancelled`	A job was cancelled: by you, sent at once; or by its spending cap, sent once the rows it delivered are billed.
`account.low_balance`	Your credit balance fell below its warning threshold, or to zero. Organization webhooks only.

Subscribe with job.* to get every job event. job.created, job.started and job.progress are accepted when you subscribe, but are not sent today.

Add a webhook

For every job of your organization: create it once. The answer includes signing_secret in full, this one time; afterwards only its first 8 characters are shown.

curl https://go-fetch.io/api/v1/webhooks/ \
  -H "X-API-Key: $GOFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-app.example.com/go-fetch/webhooks", "events": ["job.completed", "job.failed"]}'

import os

from gofetch import GoFetchClient

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])

webhook = client.webhooks().create(
    request_url="https://your-app.example.com/go-fetch/webhooks",
    event_types=["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],  # job.completed, job.failed
)
print(webhook["id"], webhook["signingSecret"])

Per-job webhooks

Add webhooks to a job when only that job should call a URL. Each one gets its own signing_secret, returned in full in the create answer under ad_hoc_webhooks, and masked afterwards.

curl https://go-fetch.io/api/v1/jobs/create/ \
  -H "X-API-Key: $GOFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "scraper_type": "instagram",
    "config": {"usernames": ["nasa"], "limit": 10},
    "webhooks": [{"url": "https://your-app.example.com/go-fetch/webhooks", "events": ["job.*"]}]
  }'

import os

from gofetch import GoFetchClient

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])

run = client.actor("instagram").start(
    run_input={"usernames": ["nasa"], "limit": 10},
    webhooks=[{
        "request_url": "https://your-app.example.com/go-fetch/webhooks",
        "event_types": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"],
    }],
)
secret = run["_gofetch_job"]["ad_hoc_webhooks"][0]["signing_secret"]
print(run["id"], len(secret))

The SDK takes Apify’s event names and translates them: ACTOR.RUN.SUCCEEDED is job.completed, ACTOR.RUN.FAILED is job.failed, ACTOR.RUN.TIMED_OUT is job.timed_out and ACTOR.RUN.ABORTED is job.cancelled. A webhook URL must use HTTPS, reach a public address (private and internal addresses are refused), be at most 2,048 characters, and not end in .php, .asp, .aspx, .jsp or .cgi.

What we send

A POST with a JSON body and these headers:

Header	Holds
`Content-Type`	`application/json`
`X-Webhook-Signature`	`sha256=` and the hex HMAC-SHA256 of the raw body, keyed with the webhook’s signing secret
`X-Event-Type`	The event, for example `job.completed`. The body does not repeat it.
`X-Delivery-ID`	The delivery’s id. Retries of a delivery keep it, so use it to ignore duplicates.

A real job.completed delivery from a test run on a local stack, headers and body. usageTotalUsd is a number; its value is shown as "…" because the docs leave amounts out.

{
  "headers": {
    "Content-Type": "application/json",
    "X-Webhook-Signature": "sha256=c445fcf129eea464b8ee7a7f1bcb04d0e7a3922aa977988a6ad773a640b46747",
    "X-Event-Type": "job.completed",
    "X-Delivery-ID": "c4873be6-9c56-4546-bbb5-b2f6f55698f0"
  },
  "body": {
    "resource": {
      "defaultDatasetId": "d1c519f1-9ae0-44c0-8362-1afe2f01a0f5",
      "id": "d1c519f1-9ae0-44c0-8362-1afe2f01a0f5",
      "status": "SUCCEEDED",
      "usageTotalUsd": "…"
    }
  }
}

The job events’ body is Apify-compatible:

Field	What it holds
`resource.id`, `resource.defaultDatasetId`	The job’s id. Read its results with `GET /api/v1/jobs/{id}/results/`.
`resource.status`	`SUCCEEDED`, `FAILED`, `TIMED-OUT` or `ABORTED`.
`resource.usageTotalUsd`	What the job was charged, in USD, as a number: the job’s `actual_cost`. `job.failed`, `job.timed_out` and the `job.cancelled` of a job you cancelled report `0`; a timed-out job whose results arrive later is charged then, and sends `job.completed` with that charge. A job stopped by its spending cap sends one `job.cancelled`, after the rows it delivered are billed, carrying that charge. If the stopped run hasn’t reported 30 minutes after the stop, the job is closed with no charge, and its `job.cancelled`, sent 30 to 35 minutes after the stop, reports `0`.

account.low_balance sends organization_id, balance, threshold, currency (USD), level (low, or zero when the balance is used up) and timestamp. Each level is sent at most once a day.

Verify the signature

Compute the HMAC-SHA256 of the raw request body, before any JSON parsing, with your signing secret, and compare it with the header in constant time. Parsing and re-serialising the body changes the bytes and breaks the check.

import hashlib
import hmac


def is_valid_signature(raw_body: bytes, signature: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected.encode(), (signature or "").encode())

import { createHmac, timingSafeEqual } from 'node:crypto';

function isValidSignature(rawBody, signature, secret) {
  const digest = createHmac('sha256', secret).update(rawBody).digest('hex');
  const expected = Buffer.from(`sha256=${digest}`);
  const received = Buffer.from(signature || '');
  return received.length === expected.length && timingSafeEqual(received, expected);
}

The SDK has the same check:

from gofetch import verify_webhook_signature


def check(raw_body: bytes, signature: str, secret: str) -> bool:
    return verify_webhook_signature(raw_body, signature, secret)

In a web app, read the raw body and the header, then reject what fails:

import os

from flask import Flask, abort, request

app = Flask(__name__)
SECRET = os.environ["GOFETCH_WEBHOOK_SECRET"]


@app.post("/go-fetch/webhooks")
def go_fetch_webhook():
    if not is_valid_signature(request.get_data(), request.headers.get("X-Webhook-Signature", ""), SECRET):
        abort(401)
    if request.headers.get("X-Event-Type") == "job.completed":
        job_id = request.get_json()["resource"]["id"]
        # Queue the job id for a worker that reads /api/v1/jobs/{id}/results/.
    return "", 204

import express from 'express';

const app = express();
const secret = process.env.GOFETCH_WEBHOOK_SECRET;

app.post('/go-fetch/webhooks', express.raw({ type: 'application/json' }), (req, res) => {
  if (!isValidSignature(req.body, req.get('X-Webhook-Signature'), secret)) return res.sendStatus(401);
  if (req.get('X-Event-Type') === 'job.completed') {
    const jobId = JSON.parse(req.body).resource.id;
    // Queue jobId for a worker that reads /api/v1/jobs/{id}/results/.
  }
  res.sendStatus(204);
});

app.listen(3000);

Answer fast, retry safely

Answer with any 2xx within 10 seconds; the connection must open within 5. Do the work afterwards, for example from a queue.
Redirects are not followed: a 3xx counts as a failure.
A failed delivery is retried up to 5 times, after 2, 4, 8, 16 and 32 minutes. A TLS certificate error is not retried.
After 5 events in a row fail every retry, the webhook is turned off: is_active becomes false and disabled_reason says why. Turn it back on with PATCH /api/v1/webhooks/{id}/ and {"is_active": true}.
The same event can arrive twice. Use X-Delivery-ID, or the job id and event, to process it once.

Test and inspect

curl -X POST https://go-fetch.io/api/v1/webhooks/$WEBHOOK_ID/test/ \
  -H "X-API-Key: $GOFETCH_API_KEY"

The test sends the event test with the body {"event": "test", "timestamp": ..., "data": {"message": ...}}, signed like any other.

curl https://go-fetch.io/api/v1/webhooks/$WEBHOOK_ID/deliveries/ \
  -H "X-API-Key: $GOFETCH_API_KEY"

Each delivery shows its event_type, status (pending, delivered or failed), response_status_code and attempts. POST /api/v1/webhooks/deliveries/{id}/retry/ sends one again.

Get started

Concepts

Scrapers

Python SDK

Events

Add a webhook

Per-job webhooks

What we send

Verify the signature

Answer fast, retry safely

Test and inspect

​Events

​Add a webhook

​Per-job webhooks

​What we send

​Verify the signature

​Answer fast, retry safely

​Test and inspect

Events

Add a webhook

Per-job webhooks

What we send

Verify the signature

Answer fast, retry safely

Test and inspect