Windmill - Scrape.do Integration

Scrape.do works in Windmill as a plain GET request from a TypeScript step, so there's nothing to install on your workers. Paste one OpenFlow YAML file and you get a flow that scrapes a list of URLs in parallel, with retries and a summary of successes and failures.

1. Create the Flow from YAML

• Sign in at app.windmill.dev or your self-hosted instance.
• Click + Flow to create a new flow.
• Open the ⋮ menu in the toolbar and choose Edit in YAML.
• Select everything in the editor and paste the contents of the flow file below.

Download this Windmill flow (OpenFlow YAML).

The graph rebuilds with two modules: scrape_loop, a for-loop that runs the scrape step, and summarize, which splits successes from failures.

2. Add Your Token and Inputs

Click Test flow. Windmill builds an input form from the flow's schema:

• token - Your API token from your dashboard. For anything you deploy or share, click the $ icon on the field and bind it to a Windmill secret variable.
• urls - The pages to scrape
• render - Turn on to render JavaScript in a headless browser
• super_proxy - Turn on to use residential and mobile proxies

render and super_proxy use extra credits, so leave them off unless the target needs them.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

3. Run the Flow

Click Test flow. The loop runs 5 iterations in parallel and retries a failed request twice, 3 seconds apart. Each iteration returns url, status, title, h1, html_length and scraped_at, and the last step returns the roll-up:

{
  "total": 3,
  "success": 3,
  "failed": 0,
  "rows": [ ... ],
  "errors": []
}

The loop settings live in the YAML: parallel: true, parallelism: 5 and skip_failures: true, so one dead URL doesn't stop the run.

4. Deploy and Trigger

• Click Deploy.
• Attach a schedule, or call the flow from its webhook URL.

Full tutorial with every step: How to Build a Parallel Web Scraper in Windmill with Scrape.do (TypeScript)