Airbyte - Scrape.do Integration | Documentation

Step-by-step guide to build a Scrape.do source connector in Airbyte Connector Builder and sync scraped pages to your warehouse

Scrape.do works in Airbyte as a custom source connector built in Connector Builder. It takes a list of URLs, scrapes each page through Scrape.do and syncs one row per page to Snowflake, BigQuery, Postgres or any other destination. Everything happens in the Airbyte UI.

1. Import the Manifest

Go to Builder in the left sidebar and click New custom connector.
Choose Import a YAML manifest and upload the manifest below.

Download the Airbyte manifest (Connector Builder YAML).

The connector has one stream, scraped_pages, configured like this:

Base URL - https://api.scrape.do
Path - /
HTTP method - GET
Authentication - No Auth, because Scrape.do takes the token as a query parameter
Query parameters - token, url, render, returnJSON, super, geoCode

render and returnJSON are fixed to true. returnJSON wraps the page in JSON, which is what Airbyte reads, and it only works with render=true. Without it, the sync finishes with an empty table.

2. Fill In the Testing Values

Open the Testing values panel and enter:

API Token - Your API token from your dashboard. It's stored encrypted and never written into the YAML.
URLs to Scrape - Two or three test URLs
Use Residential Proxies - Off, unless the target is heavily protected
Proxy Country - us by default, or another two-letter code such as gb or de

3. Test the Connector

Click Test and open the scraped_pages stream. Check that you get one record per URL, with the page HTML in content, statusCode set to 200 and source_url matching the URL that produced the row.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

4. Publish and Sync

Click the Publish chevron and select Publish to workspace.
Go to Connections > Create connection and select Scrape.do as the source.
Pick a destination and a sync frequency, then click Sync now.

Every URL is a rendered request: 5 credits on datacenter proxies and 25 with residential proxies. Size the URL list and frequency before you schedule it. See request costs.

5. Turn the HTML into Columns

Each row holds the whole page in content. Pull out the fields you need in the warehouse, for example in BigQuery:

SELECT
  source_url,
  scraped_at,
  REGEXP_EXTRACT(content, r'id="productTitle"[^>]*>\s*([^<]+?)\s*<') AS title,
  REGEXP_EXTRACT(content, r'class="a-offscreen">([^<]+)<') AS price
FROM your_dataset.scraped_pages
WHERE statusCode = 200;