# Retool - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/retool/

Step-by-step guide to integrate Scrape.do with Retool Workflows for bulk web scraping into Retool Database

Scrape.do works in Retool as a **REST API** resource. A workflow reads URLs from a table, scrapes each one through Scrape.do, parses the response and writes the results back to the same table.

## 1. Create the Scrape.do Resource

- Go to **Resources** > **Create new** > **Resource** and select **REST API**.
- Configure the resource:
    - **Name** - `Scrape.do`
    - **Base URL** - `https://api.scrape.do/`
    - **Authentication** - **None**
- Under **URL parameters**, add **Key** `token` with **Value** `{{ configVars.SCRAPE_DO_TOKEN }}` and click **Save changes**.
- Go to **Settings** > **Configuration variables**, add `SCRAPE_DO_TOKEN` and paste your API token from your [dashboard](https://dashboard.scrape.do).

The configuration variable keeps your token out of workflow exports.

## 2. Create the URL Table

In **Retool Database**, create the table the workflow reads from and writes to:

```sql
create table scrape_targets (
  id          serial primary key,
  url         text not null,
  status      text not null default 'pending',
  content     text,
  scraped_at  timestamptz
);
```

Insert a few test URLs. If your URLs already live in Postgres, MySQL, Snowflake or Google Sheets, use that resource instead.

## 3. Create the Workflow and Read the URLs

- Go to **Workflows** > **Create new** > **Workflow** and pick a **Manual** trigger for testing.
- Add a **Resource query** block named `getUrls` that runs against your database:

```sql
select id, url from scrape_targets
where status = 'pending'
limit 500
```

## 4. Loop Through the URLs with Scrape.do

- Add a **Loop** block named `scrapeLoop`:
    - **Iterable** - `{{ getUrls.data }}`
    - **Execution mode** - **Batch**
    - **Batch size** - Your Scrape.do plan concurrency
    - **Iteration delay** - `250` ms
- Inside the loop, choose the `Scrape.do` resource, set **Method** to **GET** and leave the path empty.
- Add these **URL parameters**:

| Parameter | Value |
|---|---|
| `url` | `{{ encodeURIComponent(value.url) }}` |
| `output` | `raw` to parse HTML yourself, or `markdown` for AI extraction |
| `render` | `false`, or `true` for JavaScript-heavy pages |
| `super` | `true` for residential and mobile proxies on protected targets |
| `geoCode` | A country code such as `us`, `gb` or `de`. View available locations [here](/documentation/proxies/geo-code) |

Keep `encodeURIComponent()`. Retool doesn't encode URL parameters, so a target URL with `?` or `&` breaks the request.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response.

## 5. Parse and Save the Results

- Add a **JavaScript** block named `normalize` that maps each item of `scrapeLoop.data` to a row with the matching `id` from `getUrls.data`, a `status` of `ok` or `failed`, and `scraped_at`. For sites with different layouts, use an **AI Action** block before it.
- Add a **Resource query** block named `save`:
    - **Action type** - **Bulk update via primary key**
    - **Table** - `scrape_targets`
    - **Primary key** - `id`
    - **Array of records to update** - `{{ normalize.data }}`

## 6. Test and Activate

- Click **Run** and check that the rows in `scrape_targets` move from `pending` to `ok`.
- Switch the trigger to **Schedule** and click **Enable**.

**<a href="/uploads/blog/scrapedo-retool-workflow.json" download="scrapedo-retool-workflow.json">Download this Retool workflow</a>** to import it into your organization. Importing doesn't create resources, so finish step 1 first with the exact name `Scrape.do`.

Full tutorial with every step: [How to Build a Bulk Web Scraper in Retool with Scrape.do (Low Code)](/blog/retool-web-scraper/)