# How to Build a Bulk Web Scraper in Retool with Scrape.do (Low Code)

> Source: https://scrape.do/blog/retool-web-scraper/

Published: 2026-09-07 · Updated: 2026-09-07 · Authors: Bugrahan Saka · Categories: Scraping Use Cases, Scraping Tools

Step-by-step guide to integrate Scrape.do with Retool for automated web scraping workflows

Scrape.do integrates with Retool through a REST API resource. Build a workflow that reads a list of URLs, scrapes each one through Scrape.do, parses the response with AI or JavaScript, and writes the results back to Retool Database or any other resource in your organization.

## 1. Create the Scrape.do Resource

Retool keeps connection details in resources, separate from the workflows that use them. Setting Scrape.do up as a resource means your token is stored once and never appears inside a workflow or its exported JSON.

- From your Retool dashboard, go to **Resources** and click **Create new** > **Resource**.
- Search for and select **REST API**.
- Configure the resource:
  * **Name** - Enter `Scrape.do`
  * **Base URL** - Enter `https://api.scrape.do/`
  * **Authentication** - Select **None**
- Under **URL parameters**, click **Add** and enter:
  * **Key** - `token`
  * **Value** - `{{ configVars.SCRAPE_DO_TOKEN }}`
- Click **Save changes**.

Now add the token itself. Go to **Settings** > **Configuration variables**, click **Add**, name it `SCRAPE_DO_TOKEN`, and paste your API token from the [dashboard](https://dashboard.scrape.do).

> Storing the token as a configuration variable rather than typing it into the resource keeps it out of exports and lets you use different tokens in staging and production.

## 2. Prepare Your Data Source

The workflow reads URLs from a table and writes results back to the same table. Create it in **Retool Database**:

```
create table scrape_targets (
  id          serial primary key,
  url         text not null,
  status      text not null default 'pending',
  content     text,
  scraped_at  timestamptz
);
```

Insert a few URLs to test with:

```
insert into scrape_targets (url) values
  ('https://us.amazon.com/dp/B0BLRJ4R8F'),
  ('https://us.amazon.com/dp/B0CHX1W1XY');
```

If your URLs already live in Postgres, MySQL, Snowflake, or Google Sheets, use that resource instead. Only the query in the next step changes.

## 3. Create a Workflow and Add Trigger

- From your Retool dashboard, go to **Workflows** and click **Create new** > **Workflow**.
- Click the start block to choose a trigger:
  * **Manual** - For testing and on-demand runs
  * **Schedule** - For automated hourly or daily scraping
  * **Webhook** - To trigger scraping from external sources
- Click the **+** button and add a **Resource query** block. Name it `getUrls`.
- Configure the block:
  * **Resource** - Select your database
  * **Query** - Enter the statement below

```
select id, url from scrape_targets
where status = 'pending'
limit 500
```

- Click **Run block** to confirm it returns your URLs.

## 4. Loop Through URLs with Scrape.do

A single Resource query scrapes one URL. To scrape a list, wrap it in a Loop block.

- Click the **+** button after `getUrls` and add a **Loop** block. Name it `scrapeLoop`.
- Configure the loop:
  * **Iterable** - Enter `{{ getUrls.data }}`
  * **Execution mode** - Select **Batch**
  * **Batch size** - Set to your Scrape.do plan concurrency
  * **Iteration delay** - Set to `250` ms
- Inside the loop, select **Resource** and choose `Scrape.do`.
- Configure the inner query:
  * **Method** - Select **GET**
  * **URL** - Leave the path empty, the base URL already points at the API
- Under **URL parameters**, add these one by one:
  * **url** - `{{ encodeURIComponent(value.url) }}`
  * **output** - `markdown` for AI extraction, or `raw` to parse HTML yourself
  * **render** - `false` by default, set to `true` for JavaScript heavy sites
  * **super** - `true` to use residential and mobile proxies on protected targets
  * **geoCode** - Country code for proxy location, for example `us`, `uk`, `de`

> `encodeURIComponent()` is required. Retool does not encode URL parameters for you, so a target URL containing `?` or `&` will break the request and return an error from the API. This is the single most common cause of failed requests in Retool.

- Click **Run block** and confirm the loop returns scraped content for each row.

## 5. Extract Data from Response

You now have raw HTML or markdown for every URL. There are two ways to turn that into structured data.

### Option A: Use AI to Extract Data

Best for scraping different websites with varying structures. AI returns a consistent format regardless of the source layout.

- Click the **+** button after `scrapeLoop` and add an **AI Action** block, or a **Resource query** pointed at your Anthropic or OpenAI resource.
- Configure the block:
  * **Action** - Select **Generate text**
  * **Model** - Select a model such as `claude-sonnet-4-5-20250929`
  * **Prompt** - Use the structure below

```
{{ scrapeLoop.data[0] }}

Analyze the markdown data in this document and extract ASIN, Product Name,
Product Price, Review Rating, and Review Count as a structured JSON object.
Return only the JSON, with no explanation and no code fences.
```

- Add a **JavaScript** block after it to parse the response into rows:

```js
return aiExtract.data.map((raw, i) => {
  const target = getUrls.data[i];
  let parsed = {};
  try {
    parsed = JSON.parse(raw);
  } catch (e) {
    parsed = {};
  }
  return {
    id: target.id,
    url: target.url,
    content: JSON.stringify(parsed),
    status: Object.keys(parsed).length ? "ok" : "failed",
    scraped_at: new Date().toISOString()
  };
});
```

### Option B: Use JavaScript for Extraction

If you are scraping a single site with a consistent layout, JavaScript is faster and consumes no AI credits. It also handles responses too large for a model context window.

- Click the **+** button after `scrapeLoop` and add a **JavaScript** block. Name it `normalize`.
- Paste the extraction logic:

```js
return scrapeLoop.data.map((html, i) => {
  const target = getUrls.data[i];
  const ok = html !== null && html !== undefined && html !== "";

  const pick = (re) => {
    const m = ok ? html.match(re) : null;
    return m ? m[1].replace(/\s+/g, " ").trim() : null;
  };

  return {
    id: target.id,
    url: target.url,
    name: pick(/<span id="productTitle"[^>]*>([\s\S]*?)<\/span>/),
    asin: pick(/"asin":"([A-Z0-9]{10})"/),
    price: pick(/<span class="a-offscreen">\$([0-9.,]+)<\/span>/),
    rating: pick(/(\d+\.?\d*)\s*out of/),
    status: ok ? "ok" : "failed",
    scraped_at: new Date().toISOString()
  };
});
```

- Click **Run block** to verify the fields come back populated.

## 6. Save Results to Your Database

- Click the **+** button and add a **Resource query** block. Name it `save`.
- Configure the block:
  * **Resource** - Select your database
  * **Action type** - Select **Bulk update via primary key**
  * **Table** - Select `scrape_targets`
  * **Primary key** - Select `id`
  * **Array of records to update** - Enter `{{ normalize.data }}`

Add a **Response** block at the end to return a run summary:

```js
{{ {
  total: normalize.data.length,
  ok: normalize.data.filter(r => r.status === 'ok').length,
  failed: normalize.data.filter(r => r.status === 'failed').length
} }}
```

## 7. Test and Activate

Before enabling the workflow, run it end to end.

- Click **Run** at the top of the canvas to execute every block in sequence.
- Verify that:
  * The Loop block returns content for each URL rather than nulls
  * Extraction produces populated fields, not empty strings
  * The rows in `scrape_targets` move from `pending` to `ok`

**Downloads:**

- [`scrapedo-retool-workflow.json`](/uploads/blog/scrapedo-retool-workflow.json) - the importable Retool workflow

Import it into your own Retool organization to get the exact same workflow.

> Importing a workflow does not create its resources. Complete step 1 first, using the exact resource name `Scrape.do`, or the imported blocks will have nothing to point at.

- Once the run is clean, you can:
  * Switch the trigger from **Manual** to **Schedule** and set a cron expression
  * Click **Enable** in the top right to activate the workflow
  * Use a **Webhook** trigger to scrape URLs sent from your Retool apps or external services
  * Build a Retool app on top of `scrape_targets` with a table and a button that triggers this workflow on demand