# CrewAI - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/crewai/

Step-by-step guide to integrate Scrape.do with CrewAI as a custom tool for your agents

Scrape.do integrates with CrewAI as a custom tool, `ScrapedoTool`. Every page an agent fetches goes through Scrape.do, which handles proxy rotation, anti-bot bypass, CAPTCHAs and JavaScript rendering, so your agents read the real page instead of a block screen.

## 1. Create a CrewAI Project

You need Python 3.10 to 3.13, a Scrape.do API token from your [dashboard](https://dashboard.scrape.do), and an LLM key for your agents. Scaffold a project:

```bash
crewai create crew scrapedo_crew
```

Then put your keys in the project's `.env`:

```
SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key
```

## 2. Add the Scrape.do Tool

Create `src/scrapedo_crew/tools/scrapedo_tool.py`:

```python
import os
from typing import Optional, Type

import requests
from crewai.tools import BaseTool
from pydantic import BaseModel, Field

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"

class ScrapedoToolInput(BaseModel):
    """Input schema for ScrapedoTool."""

    url: str = Field(..., description="The full URL of the page to scrape.")
    render: bool = Field(
        default=False,
        description=(
            "Set true for JavaScript-heavy pages that need a headless browser "
            "to produce the final HTML. Costs more credits."
        ),
    )
    super_proxy: bool = Field(
        default=False,
        description=(
            "Set true to route through residential/mobile proxies. Use for hard "
            "targets or after a ROTATION_FAILED error. Costs more credits."
        ),
    )

class ScrapedoTool(BaseTool):
    name: str = "scrapedo_web_scraper"
    description: str = (
        "Fetches the HTML content of a web page through the Scrape.do API. "
        "Handles proxy rotation, anti-bot systems, CAPTCHAs and JavaScript "
        "rendering server-side. Use this instead of a plain HTTP request "
        "whenever a page may be protected, rate-limited or JavaScript-rendered."
    )
    args_schema: Type[BaseModel] = ScrapedoToolInput

    api_token: Optional[str] = None
    timeout: int = 60
    max_chars: int = 100_000

    def __init__(self, api_token: Optional[str] = None, **kwargs):
        super().__init__(**kwargs)
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the tool."
            )

    def _run(self, url: str, render: bool = False, super_proxy: bool = False) -> str:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params = {"token": self.api_token, "url": url}
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            return f"Scrape.do request failed for {url}: {exc}"

        # Failed requests are not charged; see Request Costs in the docs for which statuses are.
        if response.status_code != 200:
            return (
                f"Scrape.do returned status {response.status_code} for {url}. "
                f"Response: {response.text[:500]}"
            )

        content = response.text
        if len(content) > self.max_chars:
            content = content[: self.max_chars] + "\n\n[... truncated ...]"
        return content
```

The `description` strings tell the agent when to use the tool, so keep them specific. Errors come back as strings instead of exceptions, which lets the agent retry with `render` or `super_proxy`. `max_chars` keeps a large page from filling the model's context window.

## 3. Give the Tool to an Agent

In `src/scrapedo_crew/crew.py`, add the tool to the agent that collects pages:

```python
from scrapedo_crew.tools.scrapedo_tool import ScrapedoTool

@agent
def web_data_collector(self) -> Agent:
    return Agent(
        config=self.agents_config["web_data_collector"],
        tools=[ScrapedoTool()],
        allow_delegation=False,
        verbose=True,
    )
```

In that agent's task description, spell out the escalation: start with a plain request, retry with `render` set to true if the page looks empty or JavaScript-rendered, and retry with `super_proxy` set to true if the request is blocked or returns a non-200 status.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. **Super** maps to the tool's `super_proxy` input and **Render JavaScript** to `render`.

## 4. Run the Crew

```bash
crewai install
crewai run
```

With `verbose=True`, you can watch the agent call `scrapedo_web_scraper` and work with the HTML it returns.

Full tutorial with every step: [Web Scraping in CrewAI: Give Your Agents a Tool That Actually Reads the Web](/blog/crewai-web-scraping/)