Categories:Scraping Use Cases,Scraping ToolsView as Markdown

How to Build a Parallel Web Scraper in Windmill with Scrape.do (TypeScript)

5 Mins Read

Created Date: September 07, 2026

Updated Date: September 07, 2026

Growth

Windmill turns scripts into workflows, webhooks, and cron jobs — and it's fast, claiming roughly 13x Airflow's throughput. If you're a developer, it's the automation platform that doesn't make you click through a canvas: your steps are just TypeScript or Python.

Which means scraping in Windmill starts out looking trivial. Write a fetch(), loop over URLs, done. Then you point it at a real site and get a 403, a CAPTCHA interstitial, or an HTML shell with no content because everything renders client-side. Your worker has one IP address, and target sites notice that immediately.

In this tutorial you'll build a Windmill flow that scrapes a list of URLs in parallel through Scrape.do, with retries, per-URL failure isolation, and structured output — defined entirely in OpenFlow YAML you can paste in and deploy.

What you'll build

A two-module flow:

scrape_loop — a for-loop over your URL list running 5 iterations in parallel, each firing one Scrape.do request and parsing the page.
summarize — collects the results, splitting successes from failures.

Flow inputs: your token, the URL list, and two toggles for JS rendering and residential proxies. Runs on Windmill Cloud's free tier or your own self-hosted instance.

Why Scrape.do

Scrape.do is a web scraping API: send a target URL, get the page content back. Server-side it handles what a bare fetch() can't:

Rotating proxies — datacenter by default, residential/mobile with super=true for hard targets.
Anti-bot bypass and CAPTCHA solving.
JavaScript rendering with render=true — a headless browser renders the page before it's returned.
You're only charged for successful (2xx) responses. Failed requests are free.
Free tier: 1,000 requests per month. Sign up here.

It's a plain GET request, so there's nothing to install — no scraping library, no browser binary on your workers.

Step 1 — Create the flow

Sign in at app.windmill.dev (or your self-hosted instance).
Click + Flow to create a new flow.
Open the ⋮ menu in the toolbar → Edit in YAML.
Select everything in the editor and paste in the contents of scrapedo-windmill-flow.yaml.

The graph rebuilds itself from the YAML: a for-loop containing the scrape step, followed by the summarize step.

Step 2 — Store your token as a secret

Click Test flow. Windmill auto-generates an input form from the flow's JSON Schema — token, URLs, and the two toggles.

Paste your token from the Scrape.do dashboard into the token field to test quickly. For anything you'll deploy or share, click the $ icon on that field instead and bind it to a Windmill secret variable — the token then lives encrypted in your workspace rather than in a run's arguments.

The YAML ships with no token in it, which is exactly why it's safe to share.

Step 3 — Run it

Hit Test flow. Watch the loop fan out five at a time. Each iteration returns a clean object:

{
  "url": "https://books.toscrape.com/catalogue/page-1.html",
  "status": 200,
  "title": "All products | Books to Scrape - Sandbox",
  "h1": "All products",
  "html_length": 51294,
  "scraped_at": "2026-08-18T09:14:22.117Z"
}

…and the final step returns the roll-up:

{
  "total": 3,
  "success": 3,
  "failed": 0,
  "rows": [ ... ],
  "errors": []
}

Happy with it? Deploy, then attach a schedule or webhook trigger — every Windmill flow gets a webhook URL for free, so this doubles as a scraping API endpoint for the rest of your stack.

How it works under the hood

The scrape step is ~30 lines of TypeScript, and every line of it is load-bearing:

const params = new URLSearchParams({ token, url });
if (render) params.set("render", "true");
if (super_proxy) params.set("super", "true");

const res = await fetch(`https://api.scrape.do/?${params.toString()}`, {
  signal: AbortSignal.timeout(120_000),
});

Root path only. The API responds only at https://api.scrape.do/. There is no /scrape endpoint — invented paths return access-denied.
URLSearchParams does the URL encoding. Skip encoding the target URL and Scrape.do rejects it with "Your target 'URL' is not valid."
render and super are opt-in per run, because both consume extra credits. Leave them off unless the target actually needs them.
Parsing is plain regex — no dependencies. Windmill installs npm packages automatically if you want Cheerio instead; just import it.

The parallelism and resilience live in the YAML, not the code:

type: forloopflow
parallel: true
parallelism: 5
skip_failures: true

Scraping is I/O-bound, so parallel iterations cost nothing but concurrency. skip_failures means one dead URL doesn't kill the run, and the step-level retry policy (2 attempts, 3 seconds apart) absorbs transient failures before they're ever counted as errors.

Gotchas (the real ones)

1. Transient 502 ROTATION_FAILED. A proxy hop occasionally fails on the Scrape.do side. It is not charged. The scrape step throws on non-200 precisely so the retry policy fires — throwing is what makes retries work in Windmill, swallowing the error would silently skip the retry. Persistent 502s on one domain mean the target is hard: flip super_proxy on.

2. Don't return full HTML from the loop. The step deliberately returns parsed fields plus a length counter. A 50 KB HTML string per iteration bloats the flow result, slows the UI, and makes run logs unusable. Extract first, store second.

3. skip_failures changes the shape of your results. Failed iterations still appear in the loop's result array, just not as success objects. That's why the summarize step filters on status === 200 instead of assuming every element is a clean row — assume uniform shapes here and you'll get a runtime error on the first bad URL.

4. Tune parallelism to your plan, not your ambition. Five concurrent requests is a safe default. Higher is fine for Scrape.do, but on Windmill Cloud's free tier you're sharing workers — raise parallelism only after you've watched a real run.

5. Never hardcode the token in the YAML. If you fork this flow into a repo via Git sync, an inline token gets committed. Bind it to a secret variable and keep the YAML shareable.

Run order

New flow → ⋮ → Edit in YAML → paste the template.
Test flow → enter your token (or bind a secret variable) → adjust the URL list.
Verify the output → Deploy → attach a schedule or use the auto-generated webhook.

Downloads:

scrapedo-windmill-flow.yaml - the importable Windmill flow (OpenFlow YAML)

Start scraping

Both halves are free to start: Windmill is open source (self-host it, or use the free cloud tier), and Scrape.do gives you 1,000 requests per month — proxy rotation, anti-bot bypass, CAPTCHA solving, and JavaScript rendering included, charged only on successful responses.

👉 Get your free Scrape.do API token and put the open web behind a webhook you control.

Bugrahan Saka

Growth