Categories:Scraping Use Cases,Scraping Tools

Bulk Web Scraping in Dify: Read URLs from Google Sheets and Scrape Them with Scrape.do

8 Mins Read

Created Date: July 07, 2026

Updated Date: July 07, 2026

Growth

Dify's native nodes can fetch a URL. What they can't do is get past the reasons scraping actually breaks: the target site blocks the datacenter IP, throws a CAPTCHA, or ships an empty HTML shell that only fills in once JavaScript runs. Point Dify's HTTP Request node at a protected page and you get back a challenge page or nothing at all, and no amount of node config fixes that.

This guide routes around the problem. We build a Dify Workflow that reads a list of URLs from a Google Sheet, scrapes every one through Scrape.do, and returns the results. Proxy rotation, anti-bot bypass, CAPTCHA handling, and JavaScript rendering all happen on Scrape.do's side. Your Dify workflow sends a URL and gets clean HTML back.

By the end you'll have an importable .yml you can drop into any Dify instance and run. The finished workflow file is attached at the bottom of this post.

What You'll Build

The workflow is a straight line of five nodes:

Start → HTTP Request (read the sheet) → Code (parse URLs) → Iteration (scrape each URL via Scrape.do) → Output

The HTTP Request node pulls your Google Sheet down as CSV. The Code node turns that CSV text into a clean list of URLs. The Iteration node loops over the list and calls Scrape.do once per URL. The Output node hands you back the scraped content for every page.

Everything runs on Dify's native nodes plus a single Scrape.do API call. You don't authorize a connector or install a plugin.

Why Scrape.do for This

Dify's HTTP Request node can hit a URL directly, so why route through Scrape.do at all? Because a raw request is exactly what modern sites are built to reject. The moment you scrape at any volume, you run into:

IP blocks and rate limits — the same IP hitting a site repeatedly gets throttled or banned. Scrape.do rotates through a large proxy pool automatically.
Anti-bot systems — Cloudflare and similar defenses fingerprint and challenge automated traffic. Scrape.do handles the bypass server-side.
CAPTCHAs — a hard stop for a plain request. Scrape.do solves them for you.
JavaScript-rendered pages — many sites ship an empty HTML shell and build the content with JS. A raw fetch returns nothing useful. Scrape.do's render=true runs a real headless browser and returns the fully rendered page.

Scrape.do wraps all of this behind a single GET request. You send a target URL, you get back clean HTML, and the hard parts happen on their side. If you've ever built this yourself in code, the Python web scraping guide covers what that infrastructure normally costs you to maintain.

A couple of practical details matter before we start. You're only charged for successful (2xx) responses, so a few blocked targets don't cost you anything. And the free tier includes 1,000 requests per month, which is plenty to build and validate this workflow. Grab a token on the free tier and follow along.

Prerequisites

A Dify account (cloud or self-hosted).
A free Scrape.do API token — sign up here and copy your token from the dashboard.
A Google Sheet with the URLs you want to scrape, one per row in column A.

Preparing the Google Sheet

Put your target URLs in column A, one per row, with no header:

https://books.toscrape.com/
https://quotes.toscrape.com/
https://example.com/

Then share it so the workflow can read it: Share → General access → Anyone with the link → Viewer. This lets Dify pull the sheet as CSV without any Google authentication.

Grab the sheet ID from its URL — it's the long string between /d/ and /edit:

https://docs.google.com/spreadsheets/d/THIS_IS_THE_SHEET_ID/edit

Step 1 — Create the Workflow

In Dify Studio, click Create from Blank and choose Workflow (the agentic flow option, not Chatflow — we don't need a chat interface here). Give it a name like Scrape.do Web Scraper and hit Create.

You'll land on the canvas with a Start node waiting for you. When it asks you to pick a start node, choose User Input. It's the simplest trigger, and we won't need to define any fields.

Step 2 — Read the Google Sheet (HTTP Request Node)

Add an HTTP Request node after Start. Configure it:

Method: GET
URL: your sheet's CSV export endpoint, built from the sheet ID:

https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/export?format=csv

Leave Headers, Params, and Body empty. That's the whole configuration.

Run this node on its own to confirm it works — you should get a 200 status and the CSV data for your sheet. This is the raw material the next node will parse.

Step 3 — Parse the CSV Into a URL List (Code Node)

Add a Code node after the HTTP Request. This node takes the CSV text and extracts a clean array of URLs.

Set the input variable:

Variable name: text
Value: the CSV content from the HTTP Request node's output.

Then paste this Python into the code editor:

def main(text) -> dict:
    # Normalize whatever the previous node handed us into a single string
    if isinstance(text, list):
        text = "\n".join(str(t) for t in text)

    urls = []
    for line in text.splitlines():
        line = line.strip().strip('"').strip(",")
        if line.startswith("http"):
            urls.append(line)

    return {"url_list": urls}

Set the output variable:

Name: url_list
Type: Array[String]

The logic is deliberately forgiving. It splits the CSV into lines, strips the stray quotes and commas that CSV formatting adds, and keeps only the lines that actually look like URLs. A header row or an empty trailing line won't break it. If you want a deeper walkthrough of parsing patterns like this, the Python web scraping tutorial goes further.

Run the node with a sample input to confirm it returns a clean url_list array.

Step 4 — Scrape Every URL (Iteration + Scrape.do)

This is the core of the workflow. Add an Iteration node after the Code node. Iteration loops over an array and runs whatever's inside it once per item.

Configure the Iteration node:

Input: the url_list from the Code node.
Output variable: the body from the HTTP Request node you're about to add inside the loop (we'll wire this after adding it).

Inside the Iteration block, add an HTTP Request node — this is the one that calls Scrape.do. Configure it:

Method: GET
URL: https://api.scrape.do/
Params (two rows):
- token → YOUR_SCRAPEDO_TOKEN
- url → the current iteration item (type / in the value field and select Iteration / item)

Use the Params table rather than hand-building a query string here. Dify URL-encodes param values automatically, which is exactly what Scrape.do needs for the target URL. Building the query string by hand is where these requests usually break.

Now set the Iteration node's output variable to this inner HTTP Request node's body. That tells the loop to collect the scraped HTML for each URL. The Iteration output becomes an Array[String], one scraped page per URL.

Handling Tough Targets

If a site fights back (you'll see a ROTATION_FAILED error or empty content), add one more param to the Scrape.do request:

super → true — routes through residential/mobile proxies for hard targets.

And for JavaScript-heavy pages that need a real browser:

render → true — runs the page in a headless browser and returns the fully rendered HTML.

Both are optional and cost slightly more credits, so turn them on only when a target needs them.

Step 5 — Return the Results (Output Node)

Add an Output node at the end. Set its output variable:

Name: results
Value: the Iteration node's output.

This surfaces the full array of scraped pages as the workflow's result.

Step 6 — Test End to End

Hit Test Run. The workflow will:

Pull the CSV from your sheet.
Parse it into a list of URLs.
Loop over each URL, scraping it through Scrape.do.
Return an array with the scraped content of every page.

If any node comes back empty, open the Tracing tab. It shows each node's input and output in sequence, so you can see exactly where data stops flowing and fix that node in isolation.

Step 7 — Export and Reuse

Once it's running, export the workflow as a reusable DSL file: from the app menu, choose Export DSL. This downloads a .yml that anyone can import into their own Dify instance to get the exact same workflow.

Before you share it, open the .yml in a text editor and replace your real Scrape.do token with a placeholder like YOUR_SCRAPEDO_TOKEN. Never ship a live token in a file you're publishing, and rotate it from the dashboard if it's ever been exposed.

Where to Take It Next

You've got a working bulk scraper, but the same pattern opens up a lot more. Add an LLM node after the Iteration to summarize, classify, or extract structured data from each scraped page — this is where Dify earns its keep, running scraping and reasoning in one flow. Pipe the scraped content into a Knowledge base and you have a RAG pipeline where a chatbot answers questions grounded in live web data. Or swap the Start trigger for a Schedule Trigger to re-scrape your URL list on a recurring basis.

The Scrape.do call stays identical in all of these: one GET request that quietly handles the proxies, anti-bot, and rendering so you can focus on what you do with the data.

Wrapping Up

Dify gives you the visual workflow engine; Scrape.do gives you a web that actually responds. Import the workflow below, drop in your token, point it at your own sheet, and you go from "a list of URLs in a spreadsheet" to clean scraped content for every one of them without touching a proxy config or a browser driver.

Downloads: