# How to Build a Parallel Web Scraper in Windmill with Scrape.do (TypeScript)

> Source: https://scrape.do/blog/windmill-web-scraper/

Published: 2026-09-07 · Updated: 2026-09-07 · Authors: Bugrahan Saka · Categories: Scraping Use Cases, Scraping Tools

Windmill turns scripts into workflows, webhooks, and cron jobs — and it's fast, claiming roughly 13x Airflow's throughput. If you're a developer, it's the automation platform that doesn't make you click through a canvas: your steps are just TypeScript or Python.

Which means scraping in Windmill starts out looking trivial. Write a `fetch()`, loop over URLs, done. Then you point it at a real site and get a 403, a CAPTCHA interstitial, or an HTML shell with no content because everything renders client-side. Your worker has one IP address, and target sites notice that immediately.

In this tutorial you'll build a Windmill flow that scrapes a list of URLs **in parallel** through Scrape.do, with retries, per-URL failure isolation, and structured output — defined entirely in OpenFlow YAML you can paste in and deploy.

## What you'll build

A two-module flow:

1. **`scrape_loop`** — a for-loop over your URL list running **5 iterations in parallel**, each firing one Scrape.do request and parsing the page.
2. **`summarize`** — collects the results, splitting successes from failures.

Flow inputs: your token, the URL list, and two toggles for JS rendering and residential proxies. Runs on Windmill Cloud's free tier or your own self-hosted instance.

## Why Scrape.do

Scrape.do is a web scraping API: send a target URL, get the page content back. Server-side it handles what a bare `fetch()` can't:

- **Rotating proxies** — datacenter by default, residential/mobile with `super=true` for hard targets.
- **Anti-bot bypass and CAPTCHA solving.**
- **JavaScript rendering** with `render=true` — a headless browser renders the page before it's returned.
- **You're only charged for successful (2xx) responses.** Failed requests are free.
- **Free tier: 1,000 requests per month.** [Sign up here](https://scrape.do).

It's a plain GET request, so there's nothing to install — no scraping library, no browser binary on your workers.

## Step 1 — Create the flow

1. Sign in at [app.windmill.dev](https://app.windmill.dev) (or your self-hosted instance).
2. Click **+ Flow** to create a new flow.
3. Open the **`⋮` menu** in the toolbar → **Edit in YAML**.
4. Select everything in the editor and paste in the contents of **scrapedo-windmill-flow.yaml**.

The graph rebuilds itself from the YAML: a for-loop containing the scrape step, followed by the summarize step.

## Step 2 — Store your token as a secret

Click **Test flow**. Windmill auto-generates an input form from the flow's JSON Schema — token, URLs, and the two toggles.

Paste your token from the [Scrape.do dashboard](https://dashboard.scrape.do) into the **token** field to test quickly. For anything you'll deploy or share, click the **`$` icon** on that field instead and bind it to a Windmill [secret variable](https://www.windmill.dev/docs/core_concepts/variables_and_secrets) — the token then lives encrypted in your workspace rather than in a run's arguments.

The YAML ships with no token in it, which is exactly why it's safe to share.

## Step 3 — Run it

Hit **Test flow**. Watch the loop fan out five at a time. Each iteration returns a clean object:

```json
{
  "url": "https://books.toscrape.com/catalogue/page-1.html",
  "status": 200,
  "title": "All products | Books to Scrape - Sandbox",
  "h1": "All products",
  "html_length": 51294,
  "scraped_at": "2026-08-18T09:14:22.117Z"
}
```

…and the final step returns the roll-up:

```json
{
  "total": 3,
  "success": 3,
  "failed": 0,
  "rows": [ ... ],
  "errors": []
}
```

Happy with it? **Deploy**, then attach a [schedule or webhook trigger](https://www.windmill.dev/docs/triggers) — every Windmill flow gets a webhook URL for free, so this doubles as a scraping API endpoint for the rest of your stack.

## How it works under the hood

The scrape step is ~30 lines of TypeScript, and every line of it is load-bearing:

```ts
const params = new URLSearchParams({ token, url });
if (render) params.set("render", "true");
if (super_proxy) params.set("super", "true");

const res = await fetch(`https://api.scrape.do/?${params.toString()}`, {
  signal: AbortSignal.timeout(120_000),
});
```

- **Root path only.** The API responds only at `https://api.scrape.do/`. There is no `/scrape` endpoint — invented paths return access-denied.
- **`URLSearchParams` does the URL encoding.** Skip encoding the target URL and Scrape.do rejects it with *"Your target 'URL' is not valid."*
- **`render` and `super` are opt-in per run**, because both consume extra credits. Leave them off unless the target actually needs them.
- Parsing is plain regex — no dependencies. Windmill installs npm packages automatically if you want Cheerio instead; just import it.

The parallelism and resilience live in the YAML, not the code:

```yaml
type: forloopflow
parallel: true
parallelism: 5
skip_failures: true
```

Scraping is I/O-bound, so parallel iterations cost nothing but concurrency. `skip_failures` means one dead URL doesn't kill the run, and the step-level `retry` policy (2 attempts, 3 seconds apart) absorbs transient failures before they're ever counted as errors.

## Gotchas (the real ones)

**1. Transient `502 ROTATION_FAILED`.** A proxy hop occasionally fails on the Scrape.do side. It is **not charged**. The scrape step throws on non-200 precisely so the retry policy fires — throwing is what makes retries work in Windmill, swallowing the error would silently skip the retry. Persistent 502s on one domain mean the target is hard: flip `super_proxy` on.

**2. Don't return full HTML from the loop.** The step deliberately returns parsed fields plus a length counter. A 50 KB HTML string per iteration bloats the flow result, slows the UI, and makes run logs unusable. Extract first, store second.

**3. `skip_failures` changes the shape of your results.** Failed iterations still appear in the loop's result array, just not as success objects. That's why the summarize step filters on `status === 200` instead of assuming every element is a clean row — assume uniform shapes here and you'll get a runtime error on the first bad URL.

**4. Tune parallelism to your plan, not your ambition.** Five concurrent requests is a safe default. Higher is fine for Scrape.do, but on Windmill Cloud's free tier you're sharing workers — raise `parallelism` only after you've watched a real run.

**5. Never hardcode the token in the YAML.** If you fork this flow into a repo via Git sync, an inline token gets committed. Bind it to a secret variable and keep the YAML shareable.

## Run order

1. New flow → **`⋮` → Edit in YAML** → paste the template.
2. **Test flow** → enter your token (or bind a secret variable) → adjust the URL list.
3. Verify the output → **Deploy** → attach a schedule or use the auto-generated webhook.

**Downloads:**

- [`scrapedo-windmill-flow.yaml`](/uploads/blog/scrapedo-windmill-flow.yaml) - the importable Windmill flow (OpenFlow YAML)

## Start scraping

Both halves are free to start: Windmill is open source (self-host it, or use the free cloud tier), and Scrape.do gives you **1,000 requests per month** — proxy rotation, anti-bot bypass, CAPTCHA solving, and JavaScript rendering included, charged only on successful responses.

👉 [Get your free Scrape.do API token](https://scrape.do) and put the open web behind a webhook you control.