CrewAI - Scrape.do Integration | Documentation

Step-by-step guide to integrate Scrape.do with CrewAI as a custom tool for your agents

Scrape.do integrates with CrewAI as a custom tool, ScrapedoTool. Every page an agent fetches goes through Scrape.do, which handles proxy rotation, anti-bot bypass, CAPTCHAs and JavaScript rendering, so your agents read the real page instead of a block screen.

1. Create a CrewAI Project

You need Python 3.10 to 3.13, a Scrape.do API token from your dashboard, and an LLM key for your agents. Scaffold a project:

crewai create crew scrapedo_crew

Then put your keys in the project's .env:

SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key

2. Add the Scrape.do Tool

Create src/scrapedo_crew/tools/scrapedo_tool.py:

import os
from typing import Optional, Type

import requests
from crewai.tools import BaseTool
from pydantic import BaseModel, Field

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"


class ScrapedoToolInput(BaseModel):
    """Input schema for ScrapedoTool."""

    url: str = Field(..., description="The full URL of the page to scrape.")
    render: bool = Field(
        default=False,
        description=(
            "Set true for JavaScript-heavy pages that need a headless browser "
            "to produce the final HTML. Costs more credits."
        ),
    )
    super_proxy: bool = Field(
        default=False,
        description=(
            "Set true to route through residential/mobile proxies. Use for hard "
            "targets or after a ROTATION_FAILED error. Costs more credits."
        ),
    )


class ScrapedoTool(BaseTool):
    name: str = "scrapedo_web_scraper"
    description: str = (
        "Fetches the HTML content of a web page through the Scrape.do API. "
        "Handles proxy rotation, anti-bot systems, CAPTCHAs and JavaScript "
        "rendering server-side. Use this instead of a plain HTTP request "
        "whenever a page may be protected, rate-limited or JavaScript-rendered."
    )
    args_schema: Type[BaseModel] = ScrapedoToolInput

    api_token: Optional[str] = None
    timeout: int = 60
    max_chars: int = 100_000

    def __init__(self, api_token: Optional[str] = None, **kwargs):
        super().__init__(**kwargs)
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the tool."
            )

    def _run(self, url: str, render: bool = False, super_proxy: bool = False) -> str:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params = {"token": self.api_token, "url": url}
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            return f"Scrape.do request failed for {url}: {exc}"

        # Failed requests are not charged; see Request Costs in the docs for which statuses are.
        if response.status_code != 200:
            return (
                f"Scrape.do returned status {response.status_code} for {url}. "
                f"Response: {response.text[:500]}"
            )

        content = response.text
        if len(content) > self.max_chars:
            content = content[: self.max_chars] + "\n\n[... truncated ...]"
        return content

The description strings tell the agent when to use the tool, so keep them specific. Errors come back as strings instead of exceptions, which lets the agent retry with render or super_proxy. max_chars keeps a large page from filling the model's context window.

3. Give the Tool to an Agent

In src/scrapedo_crew/crew.py, add the tool to the agent that collects pages:

from scrapedo_crew.tools.scrapedo_tool import ScrapedoTool

@agent
def web_data_collector(self) -> Agent:
    return Agent(
        config=self.agents_config["web_data_collector"],
        tools=[ScrapedoTool()],
        allow_delegation=False,
        verbose=True,
    )

In that agent's task description, spell out the escalation: start with a plain request, retry with render set to true if the page looks empty or JavaScript-rendered, and retry with super_proxy set to true if the request is blocked or returns a non-200 status.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Super maps to the tool's super_proxy input and Render JavaScript to render.