# Airbyte - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/airbyte/

Step-by-step guide to build a Scrape.do source connector in Airbyte Connector Builder and sync scraped pages to your warehouse

Scrape.do works in Airbyte as a custom source connector built in Connector Builder. It takes a list of URLs, scrapes each page through Scrape.do and syncs one row per page to Snowflake, BigQuery, Postgres or any other destination. Everything happens in the Airbyte UI.

## 1. Import the Manifest

- Go to **Builder** in the left sidebar and click **New custom connector**.
- Choose **Import a YAML manifest** and upload the manifest below.

**<a href="/uploads/blog/scrapedo-airbyte-manifest.yaml" download="scrapedo-airbyte-manifest.yaml">Download the Airbyte manifest</a>** (Connector Builder YAML).

The connector has one stream, `scraped_pages`, configured like this:

- **Base URL** - `https://api.scrape.do`
- **Path** - `/`
- **HTTP method** - `GET`
- **Authentication** - **No Auth**, because Scrape.do takes the token as a query parameter
- **Query parameters** - `token`, `url`, `render`, `returnJSON`, `super`, `geoCode`

`render` and `returnJSON` are fixed to `true`. [`returnJSON`](/documentation/headless-browser/returnjson) wraps the page in JSON, which is what Airbyte reads, and it only works with `render=true`. Without it, the sync finishes with an empty table.

## 2. Fill In the Testing Values

Open the **Testing values** panel and enter:

- **API Token** - Your API token from your [dashboard](https://dashboard.scrape.do). It's stored encrypted and never written into the YAML.
- **URLs to Scrape** - Two or three test URLs
- **Use Residential Proxies** - Off, unless the target is heavily protected
- **Proxy Country** - `us` by default, or another two-letter code such as `gb` or `de`

## 3. Test the Connector

Click **Test** and open the **scraped_pages** stream. Check that you get one record per URL, with the page HTML in `content`, `statusCode` set to `200` and `source_url` matching the URL that produced the row.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response.

## 4. Publish and Sync

- Click the **Publish** chevron and select **Publish to workspace**.
- Go to **Connections** > **Create connection** and select **Scrape.do** as the source.
- Pick a destination and a sync frequency, then click **Sync now**.

Every URL is a rendered request: 5 credits on datacenter proxies and 25 with residential proxies. Size the URL list and frequency before you schedule it. See [request costs](/documentation/request-costs).

## 5. Turn the HTML into Columns

Each row holds the whole page in `content`. Pull out the fields you need in the warehouse, for example in BigQuery:

```sql
SELECT
  source_url,
  scraped_at,
  REGEXP_EXTRACT(content, r'id="productTitle"[^>]*>\s*([^<]+?)\s*<') AS title,
  REGEXP_EXTRACT(content, r'class="a-offscreen">([^<]+)<') AS price
FROM your_dataset.scraped_pages
WHERE statusCode = 200;
```

Snowflake (`REGEXP_SUBSTR`) and Postgres (`substring ... from`) have equivalent functions.

Full tutorial with every step: [How to Build a Custom Web Scraping Connector in Airbyte with Scrape.do](/blog/airbyte-web-scraper/)