# Dify - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/dify/

Step-by-step guide to integrate Scrape.do with Dify for bulk web scraping workflows

Scrape.do integrates with Dify through the native HTTP Request node, so there's no plugin to install. The workflow below reads a list of URLs from a Google Sheet, scrapes each one through Scrape.do, and returns the content of every page.

**<a href="/uploads/blog/scrapedo-web-scraper-dify.yml" download="scrapedo-web-scraper-dify.yml">Download this Dify workflow</a>** (DSL) to import it into your Dify instance. Replace `YOUR_SCRAPEDO_TOKEN` with your token after import.

## 1. Prepare the Google Sheet

Put your URLs in column A, one per row, with no header. Then share the sheet: **Share → General access → Anyone with the link → Viewer**. This lets Dify read it as CSV without Google authentication.

Copy the sheet ID from its URL. It's the string between `/d/` and `/edit`:

```
https://docs.google.com/spreadsheets/d/THIS_IS_THE_SHEET_ID/edit
```

## 2. Create the Workflow and Read the Sheet

- In Dify Studio, click **Create from Blank**, choose **Workflow**, and give it a name.
- When asked for a start node, choose **User Input**.
- Add an **HTTP Request** node with **Method** `GET` and this **URL**:

```
https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/export?format=csv
```

Leave Headers, Params and Body empty. Run the node on its own and you should get a `200` status with your sheet's CSV.

## 3. Parse the CSV into a URL List

Add a **Code** node. Set the input variable **`text`** to the HTTP Request node's CSV output, and paste this Python:

```python
def main(text) -> dict:
    # Normalize whatever the previous node handed us into a single string
    if isinstance(text, list):
        text = "\n".join(str(t) for t in text)

    urls = []
    for line in text.splitlines():
        line = line.strip().strip('"').strip(",")
        if line.startswith("http"):
            urls.append(line)

    return {"url_list": urls}
```

Set the output variable **Name** to `url_list` and **Type** to `Array[String]`.

## 4. Scrape Every URL with Scrape.do

Add an **Iteration** node and set its **Input** to `url_list`. Inside the Iteration block, add an **HTTP Request** node:

- **Method** - `GET`
- **URL** - `https://api.scrape.do/`
- **Params** - add two rows:
    - **token** - your Scrape.do API token from your [dashboard](https://dashboard.scrape.do)
    - **url** - the current item (type `/` in the value field and select **Iteration / item**)

Use the **Params** table rather than building the query string by hand. Dify URL-encodes param values, which is what Scrape.do needs for the target URL.

Set the Iteration node's **output variable** to this inner node's `body`. The Iteration then returns an `Array[String]` with one scraped page per URL.

For hard targets, add more params to the same node:

- **super** - `true` to use residential and mobile proxies (for example after a `ROTATION_FAILED` error)
- **render** - `true` for JavaScript-heavy pages that need a headless browser

Both cost more credits, so turn them on only when a target needs them.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response.

## 5. Return the Results and Test

Add an **Output** node with an output variable named `results`, set to the Iteration node's `output`. Click **Test Run**. If a node comes back empty, open the **Tracing** tab to see each node's input and output and find where the data stops.

Full tutorial with every step: [Bulk Web Scraping in Dify: Read URLs from Google Sheets and Scrape Them with Scrape.do](/blog/dify-web-scraping/)