Python SDK

The Python SDK is the gofetch-client package on PyPI. You import it as gofetch. It keeps the interface of apify-client, so code written for Apify runs with two changed lines (Migrate from Apify). This page describes version 0.8.0.

pip install gofetch-client

It needs Python 3.9 or later. pip install "gofetch-client[async]" adds HTTP/2 for the async client.

The client

import os

from gofetch import GoFetchClient

with GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"], timeout=30.0, max_retries=3) as client:
    print(client.base_url, client.timeout)

Argument	Default	What it does
`api_key`		Your API key. `token` is accepted as another name, as in `apify-client`.
`base_url`	`https://go-fetch.io`	The API’s address.
`timeout`	`30.0`	Seconds to wait for one HTTP request.
`max_retries`	`3`	Retries for network errors, timeouts and `408`, `429`, `500`, `502`, `503`, `504`, with backoff of 1, 2 and 4 seconds. A `429` waits `Retry-After` instead.

AsyncGoFetchClient takes the same arguments; its methods are awaited, and it is used with async with.

Run a scraper

client.actor(scraper_type) picks a scraper. It also takes Apify actor names such as apify/instagram-scraper.

import os

from gofetch import GoFetchClient

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])
actor = client.actor("instagram")

# Create a job and wait for it
run = actor.call(run_input={"usernames": ["nasa"], "limit": 10}, wait_secs=600)
print(run["status"], run["isTerminal"], run["usageTotalUsd"])

# Create a job and return at once
started = actor.start(run_input={"usernames": ["natgeo"], "limit": 10})
finished = client.run(started["id"]).wait_for_finish(wait_secs=600)
print(finished["status"])

Method	What it does
`actor.call(run_input, wait_secs=None, webhooks=None)`	Creates a job and waits until it is terminal, or `wait_secs` passes, then returns the run. It does not raise when the job fails: check `run["status"]`.
`actor.start(run_input, webhooks=None)`	Creates a job and returns the run at once. `webhooks` are per-job webhooks.
`client.run(id).get()`	The run as it is now; `None` if there is no such job.
`client.run(id).wait_for_finish(wait_secs=None)`	Waits until the job is terminal. Polls every 2 seconds, backing off to 30.
`client.run(id).abort()`	Cancels the job. A finished job is returned as it is.
`client.run(id).log().get()`	The job’s log as text. `.list()` returns the entries.

run_input is sent to the API unchanged as the job’s config: use the field names in each scraper’s guide.

The run

Key	What it holds
`id`, `defaultDatasetId`	The job’s id. Both are the same.
`status`	`READY`, `RUNNING`, `SUCCEEDED`, `FAILED`, `TIMED-OUT` or `ABORTED` (the job’s `pending`, `running`, `completed`, `failed`, `timed_out`, `cancelled`).
`isTerminal`	`true` when the job has an outcome. A `FAILED` run with `isTerminal: false` is about to be retried.
`willAutoRetry`, `attemptHistory`	Whether a retry is coming, and why earlier attempts failed.
`usageTotalUsd`	The job’s charge in USD, as a number; `None` until it is settled.
`scraper_metadata`	What the scraper reported about completeness.
`startedAt`, `finishedAt`	When the job started and finished.
`_gofetch_job`	The job exactly as the API returned it.

Read results

import os

from gofetch import GoFetchClient

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])
dataset = client.dataset(os.environ["JOB_ID"])

for item in dataset.iterate_items():  # pages through every result, 100 at a time
    print(item["runId"], item.get("type"))

page = dataset.list_items(limit=50)  # a list, with Apify's page attributes
print(len(page), page.total, page.offset)

iterate_items(offset=0, limit=None) yields every result, fetching pages as it goes.
list_items(offset=0, limit=None) returns them as a list that also has items, total, offset, limit and desc.
Each item gets a runId key with the job’s id.
get_info() returns the job’s id, item count and dates. delete() does nothing: results are deleted after 5 days.

Webhooks

import os

from gofetch import GoFetchClient

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])

for webhook in client.webhooks().list()["items"]:
    print(webhook["id"], webhook["requestUrl"], webhook["eventTypes"], webhook["isActive"])

Method	What it does
`client.webhooks().list(limit=25, offset=0)`	Your organization’s webhooks.
`client.webhooks().create(request_url, event_types)`	Creates one. The answer’s `signingSecret` is in full this once.
`client.webhook(id).get()`, `.update(...)`, `.delete()`	Reads, changes (`request_url`, `event_types`, `is_active`) or deletes one.
`client.webhook(id).dispatches()`	Its deliveries.

Event names are Apify’s: ACTOR.RUN.SUCCEEDED, ACTOR.RUN.FAILED, ACTOR.RUN.TIMED_OUT, ACTOR.RUN.ABORTED. The API’s own names (job.completed, …) are accepted too. To receive them:

verify_webhook_signature(raw_body, signature, secret) checks X-Webhook-Signature (Webhooks).
transform_webhook_payload(payload, event_type) turns a delivery into Apify’s webhook shape, with eventType (pass the X-Event-Type header) and resource.usageTotalUsd.

Errors

HTTP problems raise; a job that fails does not (check run["status"]).

import os

from gofetch import (
    APIError,
    AuthenticationError,
    GoFetchClient,
    GoFetchError,
    InsufficientCreditsError,
    RateLimitError,
)

client = GoFetchClient(api_key=os.environ["GOFETCH_API_KEY"])

try:
    run = client.actor("instagram").start(run_input={"usernames": ["nasa"], "limit": 5})
    print("started", run["id"])
except AuthenticationError:
    print("The API key was refused.")
except InsufficientCreditsError as error:
    print("Not enough credit:", error.message)
except RateLimitError as error:
    print("Rate limited; retry after", error.retry_after, "seconds")
except APIError as error:
    print("Refused:", error.status_code, error.message)
except GoFetchError as error:
    print("Request failed:", error.message)

Exception	Raised on
`AuthenticationError`	`401`
`InsufficientCreditsError`	`402`
`RateLimitError`	`429`, after the retries; `retry_after` holds the seconds to wait
`APIError`	Any other error status, described by `status_code`, `error_code` and `message`; also network errors once the retries run out (`status_code` 0)
`GoFetchError`	The base class of all of them

Scraper names

client.actor() takes every scraper_type in the scraper guides. Version 0.8.0 does not list tiktok_search and tiktok_profile yet: they work, and Python prints a UserWarning you can ignore. These Apify actor names resolve to a scraper:

Apify actor	Runs
`apify/instagram-scraper`	`instagram`
`apify/instagram-profile-scraper`	`instagram_profile`
`clockworks/tiktok-profile-scraper`	`tiktok`
`streamers/youtube-scraper`	`youtube`
`apify/facebook-scraper`	`facebook`
`apify/facebook-posts-scraper`	`facebook_posts`
`apify/facebook-pages-scraper`	`facebook_profile`
`scraperlink/google-search-results-serp-scraper`	`google_serp`

Get started

Concepts

Scrapers

Python SDK

The client

Run a scraper

The run

Read results

Webhooks

Errors

Scraper names

​The client

​Run a scraper

​The run

​Read results

​Webhooks

​Errors

​Scraper names

The client

Run a scraper

The run

Read results

Webhooks

Errors

Scraper names