Dify - Scrape.do Integration

Scrape.do integrates with Dify through the native HTTP Request node, so there's no plugin to install. The workflow below reads a list of URLs from a Google Sheet, scrapes each one through Scrape.do, and returns the content of every page.

Download this Dify workflow (DSL) to import it into your Dify instance. Replace YOUR_SCRAPEDO_TOKEN with your token after import.

1. Prepare the Google Sheet

Put your URLs in column A, one per row, with no header. Then share the sheet: Share → General access → Anyone with the link → Viewer. This lets Dify read it as CSV without Google authentication.

Copy the sheet ID from its URL. It's the string between /d/ and /edit:

https://docs.google.com/spreadsheets/d/THIS_IS_THE_SHEET_ID/edit

2. Create the Workflow and Read the Sheet

• In Dify Studio, click Create from Blank, choose Workflow, and give it a name.
• When asked for a start node, choose User Input.
• Add an HTTP Request node with Method GET and this URL:

https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/export?format=csv

Leave Headers, Params and Body empty. Run the node on its own and you should get a 200 status with your sheet's CSV.

3. Parse the CSV into a URL List

Add a Code node. Set the input variable text to the HTTP Request node's CSV output, and paste this Python:

def main(text) -> dict:
    # Normalize whatever the previous node handed us into a single string
    if isinstance(text, list):
        text = "\n".join(str(t) for t in text)

    urls = []
    for line in text.splitlines():
        line = line.strip().strip('"').strip(",")
        if line.startswith("http"):
            urls.append(line)

    return {"url_list": urls}

Set the output variable Name to url_list and Type to Array[String].

4. Scrape Every URL with Scrape.do

Add an Iteration node and set its Input to url_list. Inside the Iteration block, add an HTTP Request node:

• Method - GET
• URL - https://api.scrape.do/
• Params - add two rows:
  • token - your Scrape.do API token from your dashboard
  • url - the current item (type / in the value field and select Iteration / item)

Use the Params table rather than building the query string by hand. Dify URL-encodes param values, which is what Scrape.do needs for the target URL.

Set the Iteration node's output variable to this inner node's body. The Iteration then returns an Array[String] with one scraped page per URL.

For hard targets, add more params to the same node:

• super - true to use residential and mobile proxies (for example after a ROTATION_FAILED error)
• render - true for JavaScript-heavy pages that need a headless browser

Both cost more credits, so turn them on only when a target needs them.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

5. Return the Results and Test

Add an Output node with an output variable named results, set to the Iteration node's output. Click Test Run. If a node comes back empty, open the Tracing tab to see each node's input and output and find where the data stops.

Full tutorial with every step: Bulk Web Scraping in Dify: Read URLs from Google Sheets and Scrape Them with Scrape.do