Retool - Scrape.do Integration

Scrape.do works in Retool as a REST API resource. A workflow reads URLs from a table, scrapes each one through Scrape.do, parses the response and writes the results back to the same table.

1. Create the Scrape.do Resource

• Go to Resources > Create new > Resource and select REST API.
• Configure the resource:
  • Name - Scrape.do
  • Base URL - https://api.scrape.do/
  • Authentication - None
• Under URL parameters, add Key token with Value {{ configVars.SCRAPE_DO_TOKEN }} and click Save changes.
• Go to Settings > Configuration variables, add SCRAPE_DO_TOKEN and paste your API token from your dashboard.

The configuration variable keeps your token out of workflow exports.

2. Create the URL Table

In Retool Database, create the table the workflow reads from and writes to:

create table scrape_targets (
  id          serial primary key,
  url         text not null,
  status      text not null default 'pending',
  content     text,
  scraped_at  timestamptz
);

Insert a few test URLs. If your URLs already live in Postgres, MySQL, Snowflake or Google Sheets, use that resource instead.

3. Create the Workflow and Read the URLs

• Go to Workflows > Create new > Workflow and pick a Manual trigger for testing.
• Add a Resource query block named getUrls that runs against your database:

select id, url from scrape_targets
where status = 'pending'
limit 500

4. Loop Through the URLs with Scrape.do

• Add a Loop block named scrapeLoop:
  • Iterable - {{ getUrls.data }}
  • Execution mode - Batch
  • Batch size - Your Scrape.do plan concurrency
  • Iteration delay - 250 ms
• Inside the loop, choose the Scrape.do resource, set Method to GET and leave the path empty.
• Add these URL parameters:

Keep encodeURIComponent(). Retool doesn't encode URL parameters, so a target URL with ? or & breaks the request.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

5. Parse and Save the Results

• Add a JavaScript block named normalize that maps each item of scrapeLoop.data to a row with the matching id from getUrls.data, a status of ok or failed, and scraped_at. For sites with different layouts, use an AI Action block before it.
• Add a Resource query block named save:
  • Action type - Bulk update via primary key
  • Table - scrape_targets
  • Primary key - id
  • Array of records to update - {{ normalize.data }}

6. Test and Activate

• Click Run and check that the rows in scrape_targets move from pending to ok.
• Switch the trigger to Schedule and click Enable.

Download this Retool workflow to import it into your organization. Importing doesn't create resources, so finish step 1 first with the exact name Scrape.do.

Full tutorial with every step: How to Build a Bulk Web Scraper in Retool with Scrape.do (Low Code)