# Web Scraping in CrewAI: Give Your Agents a Tool That Actually Reads the Web

> Source: https://scrape.do/blog/crewai-web-scraping/

Published: 2026-07-13 · Updated: 2026-07-13 · Authors: Bugrahan Saka · Categories: Scraping Use Cases, Scraping Tools

An agent that can't tell a real page from a block page is worse than an agent that can't browse at all. The one that can't browse fails loudly. The one holding a broken tool fails quietly: it fetches a Cloudflare challenge screen, reads the headings and body text sitting right there in the HTML, and writes you a confident summary of a page it never actually saw.

CrewAI gives every agent a role, a goal, and a set of tools, then lets the crew work the problem together. That model is clean until the moment an agent reaches for the open web, because from that point on the tool you handed it *is* the agent's entire perception of the page. Hand it `requests.get()` and it perceives whatever a datacenter IP gets served, which on anything commercial is a fingerprinted block, a rate-limit wall, a CAPTCHA, or an empty shell that JavaScript was supposed to fill.

So this tutorial isn't really about crews. It's about the one file that decides whether your agents see the web or hallucinate it. We'll build **`ScrapedoTool`**, a CrewAI custom tool that routes every fetch through [Scrape.do](https://scrape.do), which handles proxy rotation, anti-bot bypass, CAPTCHA solving, and JavaScript rendering on its side. Then we'll prove it out with a small two-agent crew. Write the tool once and every agent in every crew you build afterward inherits a web that answers.

## The failure mode you're actually fixing

Naive scraping breaks in four ways, and each one lands on the exact spot agents are weakest.

An agent doesn't experience an **IP ban** as a ban. It experiences it as a page, reads that page, and moves on. It doesn't experience an **anti-bot challenge** as a challenge, that Cloudflare interstitial has a title and paragraphs, and the agent will summarize them as if they were the article. A **CAPTCHA** is a hard stop for a raw request, but the agent only learns that if the tool tells it. And a **JavaScript-rendered page** comes back as valid markup with no content in it, so the agent reports the page is empty, or worse, invents what it assumes should have been there.

Every one of these is a case where the request technically succeeded and the *content* is garbage. That's the dangerous class of failure for an autonomous system, and it's exactly what routing through Scrape.do removes: one GET request in, the real rendered page out, no challenge screens leaking into your agent's context.

Two things about that arrangement matter specifically for agents, which retry more than any human ever would:

- **Failed requests are free.** You're only billed for successful 2xx responses, so an agent hammering a stubborn target through a retry loop doesn't run up a bill on the attempts that bounce.
- **The free tier is 1,000 requests a month**, enough to build and stress-test a crew before you pay for anything. [Grab a token here.](https://scrape.do)

## What we'll build

A sequential two-agent crew that turns a URL into a structured report:

- A **Web Data Collector** fetches the target through Scrape.do, escalating to JavaScript rendering or residential proxies when a page fights back.
- A **Content Analyst** reads the retrieved HTML and writes a clean summary: title, topic, key points, notable items.

The crew is the demo. `ScrapedoTool` is the deliverable, once it exists, any agent you write can reach any page.

## Prerequisites

- Python >=3.10, <3.14
- A CrewAI project scaffolded with `crewai create crew scrapedo_crew`
- A free Scrape.do token from [scrape.do](https://scrape.do)
- An LLM key for your agents (`OPENAI_API_KEY` or equivalent)

## Building the tool

Create `src/scrapedo_crew/tools/scrapedo_tool.py`. This is the whole integration; every later step is just wiring it into a crew.

```python
"""Scrape.do custom tool for CrewAI.

Wraps the Scrape.do API so an agent can fetch any web page with proxy
rotation, anti-bot bypass, CAPTCHA handling and optional JS rendering.
"""

import os
from typing import Optional, Type
from urllib.parse import quote

import requests
from crewai.tools import BaseTool
from pydantic import BaseModel, Field

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"


class ScrapedoToolInput(BaseModel):
    """Input schema for ScrapedoTool."""

    url: str = Field(..., description="The full URL of the page to scrape.")
    render: bool = Field(
        default=False,
        description=(
            "Set true for JavaScript-heavy pages that need a headless browser "
            "to produce the final HTML. Costs more credits."
        ),
    )
    super_proxy: bool = Field(
        default=False,
        description=(
            "Set true to route through residential/mobile proxies. Use for hard "
            "targets or after a ROTATION_FAILED error. Costs more credits."
        ),
    )


class ScrapedoTool(BaseTool):
    name: str = "scrapedo_web_scraper"
    description: str = (
        "Fetches the HTML content of a web page through the Scrape.do API. "
        "Handles proxy rotation, anti-bot systems, CAPTCHAs and JavaScript "
        "rendering server-side. Use this instead of a plain HTTP request "
        "whenever a page may be protected, rate-limited or JavaScript-rendered."
    )
    args_schema: Type[BaseModel] = ScrapedoToolInput

    api_token: Optional[str] = None
    timeout: int = 60
    max_chars: int = 100_000

    def __init__(self, api_token: Optional[str] = None, **kwargs):
        super().__init__(**kwargs)
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the tool."
            )

    def _run(self, url: str, render: bool = False, super_proxy: bool = False) -> str:
        params = {
            "token": self.api_token,
            # Scrape.do requires the target URL to be encoded.
            "url": quote(url, safe=""),
        }
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            return f"Scrape.do request failed for {url}: {exc}"

        # Scrape.do only charges for successful (2xx) responses.
        if response.status_code != 200:
            return (
                f"Scrape.do returned status {response.status_code} for {url}. "
                f"Response: {response.text[:500]}"
            )

        content = response.text
        if len(content) > self.max_chars:
            content = content[: self.max_chars] + "\n\n[... truncated ...]"
        return content
```

Short as it is, four decisions in that file are load-bearing, and getting any of them wrong is the difference between a tool that works and one that quietly poisons the agent's context.

The **description is written for the model, not for you**. CrewAI feeds `description` and every field description straight into the agent's context; that text *is* how the agent decides when to reach for the tool and how to call it. "Use this instead of a plain HTTP request whenever a page may be protected, rate-limited or JavaScript-rendered" is a direct instruction to the model. A vague description is the single most common reason an agent ignores a tool you gave it and improvises with something worse.

`render` and `super_proxy` **live in the input schema**, which puts the escalation decision in the agent's hands per request. Both cost extra credits, so both default off, and the agent flips them on when a page demands it. Judging "this looks like it needs a real browser" is exactly the kind of call agents are good at.

The target URL is **encoded with `quote(url, safe="")`** because Scrape.do expects an encoded target. Skip it and any URL carrying its own query string breaks in ways that look like the site's fault rather than an encoding bug.

And **errors come back as strings, not exceptions**. A raised exception ends the agent's turn; a returned error string tells the agent what went wrong and lets it decide the next move, retry with rendering, escalate to residential proxies, or report the failure honestly. That single choice is what makes the escalation ladder in the task description below actually function. The `max_chars` ceiling is the fifth quiet guard: raw HTML is enormous, and an untruncated content-heavy page will blow past your model's context window or run up a surprising token bill.

## Wiring the crew around it

With the tool built, the rest is configuration. Define the two agents in `src/scrapedo_crew/config/agents.yaml`:

```yaml
web_data_collector:
  role: >
    Web Data Collector
  goal: >
    Retrieve the full, unblocked content of the target URL: {url}
  backstory: >
    You are a specialist at pulling web pages that fight back — pages behind
    anti-bot systems, rate limits, CAPTCHAs, or built entirely with JavaScript.
    You always use the Scrape.do tool to fetch content, and you never fabricate
    data you did not actually retrieve. If a page comes back empty or looks
    JavaScript-driven, you retry with rendering enabled. If a request is blocked
    or rotation fails, you retry through residential proxies.

content_analyst:
  role: >
    Content Analyst
  goal: >
    Turn raw scraped HTML into a clean, structured summary a human can act on.
  backstory: >
    You read raw HTML and extract what matters — the page title, the main topic,
    the key facts, and any prices, products or named entities present. You work
    strictly from the content you were given and never invent details that are
    not in the source.
```

The anti-hallucination language in both backstories isn't filler. "You never fabricate data you did not actually retrieve" and "never invent details that are not in the source" are the guardrail against the precise failure this whole post is about, an agent narrating a block page or filling in content it never received.

Now the tasks, in `src/scrapedo_crew/config/tasks.yaml`:

```yaml
scrape_page_task:
  description: >
    Scrape the following URL and return its full content: {url}

    Use the Scrape.do web scraper tool. Start with a plain request.
    If the returned page looks empty, truncated, or is clearly rendered by
    JavaScript, retry the same URL with render set to true.
    If the request is blocked, returns a non-200 status, or reports a rotation
    failure, retry with super_proxy set to true.
    Return the retrieved content exactly as received.
  expected_output: >
    The raw HTML content of the page at {url}.
  agent: web_data_collector

analyze_content_task:
  description: >
    Using the scraped content from the previous task, produce a structured
    summary of the page. Work only from the content that was actually
    retrieved — do not add information from your own knowledge.
  expected_output: >
    A markdown summary with exactly these sections:

    ## Title
    The page title.

    ## Topic
    One or two sentences describing what the page is about.

    ## Key Points
    Three to five bullet points covering the most important facts.

    ## Notable Items
    Any products, prices, or named entities found on the page.
    Write "None found" if there are none.
  agent: content_analyst
  output_file: report.md
```

The escalation ladder lives in the task description, plain request, then `render`, then `super_proxy`. It only works *because* the tool returns errors as readable strings: the agent sees the failure text and knows which rung to climb next. `output_file: report.md` drops the analyst's summary straight to disk.

Finally, assemble the crew in `src/scrapedo_crew/crew.py`:

```python
from typing import List

from crewai import LLM
from crewai import Agent, Crew, Process, Task
from crewai.agents.agent_builder.base_agent import BaseAgent
from crewai.project import CrewBase, agent, crew, task

from scrapedo_crew.tools.scrapedo_tool import ScrapedoTool


@CrewBase
class ScrapedoCrew:
    """Scrape.do web scraping crew."""

    agents: List[BaseAgent]
    tasks: List[Task]

    agents_config = "config/agents.yaml"
    tasks_config = "config/tasks.yaml"

    @agent
    def web_data_collector(self) -> Agent:
        return Agent(
            config=self.agents_config["web_data_collector"],
            tools=[ScrapedoTool()],
            allow_delegation=False,
            verbose=True,
        )

    @agent
    def content_analyst(self) -> Agent:
        return Agent(
            config=self.agents_config["content_analyst"],
            allow_delegation=False,
            verbose=True,
        )

    @task
    def scrape_page_task(self) -> Task:
        return Task(
            config=self.tasks_config["scrape_page_task"],
        )

    @task
    def analyze_content_task(self) -> Task:
        return Task(
            config=self.tasks_config["analyze_content_task"],
            context=[self.scrape_page_task()],
        )

    @crew
    def crew(self) -> Crew:
        return Crew(
            agents=self.agents,
            tasks=self.tasks,
            process=Process.sequential,
            verbose=True,
            chat_llm=LLM(model="openai/gpt-4o"),
        )
```

`ScrapedoTool()` goes in the collector's `tools` list and nowhere else, the analyst carries no tools because it only reasons over what it's handed. `context=[self.scrape_page_task()]` passes the scraped content forward.

## Running it

Set the inputs in `src/scrapedo_crew/main.py`:

```python
DEFAULT_INPUTS = {
    "url": "https://books.toscrape.com/",
}


def run():
    ScrapedoCrew().crew().kickoff(inputs=DEFAULT_INPUTS)
```

Drop your keys in `.env`:

```
SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key
```

Then:

```bash
crewai install
crewai run
```

With `verbose=True` you'll watch the collector call `scrapedo_web_scraper`, then the analyst reason over the HTML it returned. The structured summary lands in `report.md`.

## Scaling to a list of URLs

For a batch, kick the crew off once per URL. Each run is independent, so one bad page doesn't take down the rest:

```python
BATCH_URLS = [
    "https://books.toscrape.com/",
    "https://quotes.toscrape.com/",
]


def run_batch():
    results = {}
    for url in BATCH_URLS:
        print(f"\n{'=' * 60}\nProcessing: {url}\n{'=' * 60}")
        try:
            results[url] = ScrapedoCrew().crew().kickoff(inputs={"url": url})
        except Exception as e:
            print(f"Failed on {url}: {e}")
            results[url] = None
    return results
```

Because only successful responses are billed, a batch with a few unreachable targets costs you just the pages you actually got back.

## Scaffolding it visually in Crew Studio

On CrewAI Enterprise, **Crew Studio** gives you a visual canvas for the same structure: describe the automation in chat and it scaffolds agents, tasks, and tools as editable nodes. It's a fast way to shape a crew, and it hands you a downloadable project.

The Scrape.do tool still comes from this post, though. Scaffold the crew in Studio, download the code, drop `scrapedo_tool.py` into `src/<your_project>/tools/`, and add `tools=[ScrapedoTool()]` to the collector agent. Keep your token in a `SCRAPEDO_TOKEN` environment variable rather than pasting it into a node.

## Things that will bite you

**Encode the target URL.** `quote(url, safe="")` is not optional. Targets with query strings fail in ways that look like site problems, not encoding problems.

**Mind the context window.** Raw HTML is huge. `max_chars` exists so one content-heavy page can't exhaust your model's context or spike your bill.

**If the agent won't use the tool, fix the description.** When a collector ignores `scrapedo_web_scraper`, the answer is almost always in the `description` string, not the code. Spell out when to use it.

**Escalate on purpose.** `render=true` and `super=true` both cost more. Leave them off by default and let the agent turn them on when a page truly needs a browser or a residential IP, which is what the task's escalation ladder is for.

**Keep the token out of your code.** Read it from the environment. Never commit it, never paste it into a screenshot or an exported project.

## Where to take it next

The tool is the reusable piece; the crew around it is interchangeable. A few directions:

- **Add a parser agent** between collector and analyst that extracts fields into a Pydantic model with `output_pydantic`, and you get typed data instead of prose.
- **Feed a RAG pipeline** by piping scraped content into a knowledge source so a downstream crew answers questions grounded in live web data.
- **Hand it to any research crew** doing competitive monitoring, price tracking, or lead enrichment, they all take the same `tools=[ScrapedoTool()]` line.

## Wrapping up

CrewAI gives your agents reasoning and collaboration. What it can't give them on its own is a web that answers instead of one that blocks, challenges, and lies by omission. That's one tool file, and once it's in your project, every agent you write reaches any page without you ever touching a proxy config or a browser driver.

[Get a free Scrape.do token](https://scrape.do), 1,000 requests a month, no credit card, and give your crew the open web.

**Downloads:**

- [`scrapedo-crew.zip`](/uploads/blog/scrapedo-crew.zip) — the complete CrewAI project from this post