Categories:Scraping Use Cases,Scraping ToolsView as Markdown

How to Build a Bulk Web Scraper in Retool with Scrape.do (Low Code)

Clock6 Mins Read
calendarCreated Date: September 07, 2026
calendarUpdated Date: September 07, 2026

Step-by-step guide to integrate Scrape.do with Retool for automated web scraping workflows

Scrape.do integrates with Retool through a REST API resource. Build a workflow that reads a list of URLs, scrapes each one through Scrape.do, parses the response with AI or JavaScript, and writes the results back to Retool Database or any other resource in your organization.

1. Create the Scrape.do Resource

Retool keeps connection details in resources, separate from the workflows that use them. Setting Scrape.do up as a resource means your token is stored once and never appears inside a workflow or its exported JSON.

  • From your Retool dashboard, go to Resources and click Create new > Resource.
  • Search for and select REST API.
  • Configure the resource:
    • Name - Enter Scrape.do
    • Base URL - Enter https://api.scrape.do/
    • Authentication - Select None
  • Under URL parameters, click Add and enter:
    • Key - token
    • Value - {{ configVars.SCRAPE_DO_TOKEN }}
  • Click Save changes.

Now add the token itself. Go to Settings > Configuration variables, click Add, name it SCRAPE_DO_TOKEN, and paste your API token from the dashboard.

Storing the token as a configuration variable rather than typing it into the resource keeps it out of exports and lets you use different tokens in staging and production.

2. Prepare Your Data Source

The workflow reads URLs from a table and writes results back to the same table. Create it in Retool Database:

create table scrape_targets (
  id          serial primary key,
  url         text not null,
  status      text not null default 'pending',
  content     text,
  scraped_at  timestamptz
);

Insert a few URLs to test with:

insert into scrape_targets (url) values
  ('https://us.amazon.com/dp/B0BLRJ4R8F'),
  ('https://us.amazon.com/dp/B0CHX1W1XY');

If your URLs already live in Postgres, MySQL, Snowflake, or Google Sheets, use that resource instead. Only the query in the next step changes.

3. Create a Workflow and Add Trigger

  • From your Retool dashboard, go to Workflows and click Create new > Workflow.
  • Click the start block to choose a trigger:
    • Manual - For testing and on-demand runs
    • Schedule - For automated hourly or daily scraping
    • Webhook - To trigger scraping from external sources
  • Click the + button and add a Resource query block. Name it getUrls.
  • Configure the block:
    • Resource - Select your database
    • Query - Enter the statement below
select id, url from scrape_targets
where status = 'pending'
limit 500
  • Click Run block to confirm it returns your URLs.

4. Loop Through URLs with Scrape.do

A single Resource query scrapes one URL. To scrape a list, wrap it in a Loop block.

  • Click the + button after getUrls and add a Loop block. Name it scrapeLoop.
  • Configure the loop:
    • Iterable - Enter {{ getUrls.data }}
    • Execution mode - Select Batch
    • Batch size - Set to your Scrape.do plan concurrency
    • Iteration delay - Set to 250 ms
  • Inside the loop, select Resource and choose Scrape.do.
  • Configure the inner query:
    • Method - Select GET
    • URL - Leave the path empty, the base URL already points at the API
  • Under URL parameters, add these one by one:
    • url - {{ encodeURIComponent(value.url) }}
    • output - markdown for AI extraction, or raw to parse HTML yourself
    • render - false by default, set to true for JavaScript heavy sites
    • super - true to use residential and mobile proxies on protected targets
    • geoCode - Country code for proxy location, for example us, uk, de

encodeURIComponent() is required. Retool does not encode URL parameters for you, so a target URL containing ? or & will break the request and return an error from the API. This is the single most common cause of failed requests in Retool.

  • Click Run block and confirm the loop returns scraped content for each row.

5. Extract Data from Response

You now have raw HTML or markdown for every URL. There are two ways to turn that into structured data.

Option A: Use AI to Extract Data

Best for scraping different websites with varying structures. AI returns a consistent format regardless of the source layout.

  • Click the + button after scrapeLoop and add an AI Action block, or a Resource query pointed at your Anthropic or OpenAI resource.
  • Configure the block:
    • Action - Select Generate text
    • Model - Select a model such as claude-sonnet-4-5-20250929
    • Prompt - Use the structure below
{{ scrapeLoop.data[0] }}

Analyze the markdown data in this document and extract ASIN, Product Name,
Product Price, Review Rating, and Review Count as a structured JSON object.
Return only the JSON, with no explanation and no code fences.
  • Add a JavaScript block after it to parse the response into rows:
return aiExtract.data.map((raw, i) => {
  const target = getUrls.data[i];
  let parsed = {};
  try {
    parsed = JSON.parse(raw);
  } catch (e) {
    parsed = {};
  }
  return {
    id: target.id,
    url: target.url,
    content: JSON.stringify(parsed),
    status: Object.keys(parsed).length ? "ok" : "failed",
    scraped_at: new Date().toISOString()
  };
});

Option B: Use JavaScript for Extraction

If you are scraping a single site with a consistent layout, JavaScript is faster and consumes no AI credits. It also handles responses too large for a model context window.

  • Click the + button after scrapeLoop and add a JavaScript block. Name it normalize.
  • Paste the extraction logic:
return scrapeLoop.data.map((html, i) => {
  const target = getUrls.data[i];
  const ok = html !== null && html !== undefined && html !== "";

  const pick = (re) => {
    const m = ok ? html.match(re) : null;
    return m ? m[1].replace(/\s+/g, " ").trim() : null;
  };

  return {
    id: target.id,
    url: target.url,
    name: pick(/<span id="productTitle"[^>]*>([\s\S]*?)<\/span>/),
    asin: pick(/"asin":"([A-Z0-9]{10})"/),
    price: pick(/<span class="a-offscreen">\$([0-9.,]+)<\/span>/),
    rating: pick(/(\d+\.?\d*)\s*out of/),
    status: ok ? "ok" : "failed",
    scraped_at: new Date().toISOString()
  };
});
  • Click Run block to verify the fields come back populated.

6. Save Results to Your Database

  • Click the + button and add a Resource query block. Name it save.
  • Configure the block:
    • Resource - Select your database
    • Action type - Select Bulk update via primary key
    • Table - Select scrape_targets
    • Primary key - Select id
    • Array of records to update - Enter {{ normalize.data }}

Add a Response block at the end to return a run summary:

{{ {
  total: normalize.data.length,
  ok: normalize.data.filter(r => r.status === 'ok').length,
  failed: normalize.data.filter(r => r.status === 'failed').length
} }}

7. Test and Activate

Before enabling the workflow, run it end to end.

  • Click Run at the top of the canvas to execute every block in sequence.
  • Verify that:
    • The Loop block returns content for each URL rather than nulls
    • Extraction produces populated fields, not empty strings
    • The rows in scrape_targets move from pending to ok

Downloads:

Import it into your own Retool organization to get the exact same workflow.

Importing a workflow does not create its resources. Complete step 1 first, using the exact resource name Scrape.do, or the imported blocks will have nothing to point at.

  • Once the run is clean, you can:
    • Switch the trigger from Manual to Schedule and set a cron expression
    • Click Enable in the top right to activate the workflow
    • Use a Webhook trigger to scrape URLs sent from your Retool apps or external services
    • Build a Retool app on top of scrape_targets with a table and a button that triggers this workflow on demand
Table of Contents

Reliable, Scalable, Unstoppable Web Scraping

START FOR FREE