Categories:Scraping Use Cases,Scraping Tools

Web Scraping in LlamaIndex: A Workflow That Reads the Live Web with Scrape.do

9 Mins Read

Created Date: July 16, 2026

Updated Date: July 16, 2026

Growth

The dangerous failure in a RAG pipeline is the one that doesn't raise an exception. A 500 error you'll catch. A block page you won't, because it arrives as a perfectly valid HTTP 200 with a title, some headings, and a paragraph explaining that access is denied. LlamaIndex chunks it, embeds it, stores it, and retrieves it later as if it were the document you wanted. Your LLM then answers a user's question from a Cloudflare challenge screen. Garbage in, confident garbage out, and nothing anywhere logged an error.

LlamaIndex has always been the framework for pointing an LLM at your data. In 2026 that data increasingly lives on the open web rather than in a tidy folder of PDFs, and the built-in web readers fetch a URL and hand you back whatever came down the wire. Against a static docs page that's fine. Against anything commercial it's a challenge screen or an empty JavaScript shell, and your index gets built on it.

This tutorial closes that gap end to end. We'll write a ScrapedoReader that loads any URL as a clean LlamaIndex Document through Scrape.do, then wire it into an event-driven Workflow that scrapes, indexes, and answers questions about live pages, with the slow, failure-prone scraping stage cleanly isolated from the retrieval logic. Scrape.do handles the proxies, anti-bot bypass, CAPTCHA solving, and JS rendering; LlamaIndex handles the reasoning.

Why a plain web reader isn't enough

A raw fetch is exactly what modern sites are built to reject, and for a RAG pipeline the consequences are worse than a crash:

IP blocks and rate limits. Hit a site repeatedly from one IP and you get throttled, then banned. Scrape.do rotates a large proxy pool automatically.
Anti-bot systems. Cloudflare-style defenses fingerprint automated traffic and return a challenge page, real text that a naive reader indexes as though it were content.
CAPTCHAs. A hard stop for a plain request. Scrape.do solves them server-side.
JavaScript-rendered pages. Many sites ship an empty shell and build the body client-side. A raw fetch returns markup with nothing in it. Scrape.do's render=true runs a real headless browser and returns the fully rendered page.

The through-line is silence. None of these throw. Each one becomes a plausible-looking document in your index, so the only defense is to make sure the documents entering the index are the real pages in the first place. That's what routing through Scrape.do buys you.

Two practical notes before we build: you're only charged for successful (2xx) responses, so the reader skipping a few unreachable URLs in a batch costs nothing, and the free tier is 1,000 requests a month, enough to build and test a real pipeline. Grab a token here.

The two pieces

ScrapedoReader — a data loader that turns a list of URLs into LlamaIndex Document objects. Usable on its own in any LlamaIndex project.
ScrapedoRAGWorkflow — an event-driven Workflow that chains three typed stages into a full retrieval pipeline:

StartEvent → [scrape] → ScrapedEvent → [index] → IndexedEvent → [query] → StopEvent

The Workflow is the architecturally interesting half, so it's worth understanding why it's shaped this way before we get there.

Prerequisites

Python >=3.10
pip install llama-index llama-index-llms-openai requests python-dotenv
A free Scrape.do token from scrape.do
An OPENAI_API_KEY (used for embeddings and answering)

The reader

This is the integration. Everything after it is standard LlamaIndex. Create scrapedo_llamaindex/reader.py:

"""Scrape.do reader for LlamaIndex.

Fetches web pages through the Scrape.do API — with proxy rotation, anti-bot
bypass, CAPTCHA solving and optional JS rendering — and returns them as
LlamaIndex Document objects ready for indexing.
"""

import os
from typing import List, Optional
from urllib.parse import quote

import requests
from llama_index.core.schema import Document

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"


class ScrapedoReader:
    """Load web pages as LlamaIndex Documents via the Scrape.do API."""

    def __init__(self, api_token: Optional[str] = None, timeout: int = 60):
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the reader."
            )
        self.timeout = timeout

    def _scrape(self, url: str, render: bool, super_proxy: bool) -> Optional[str]:
        params = {
            "token": self.api_token,
            # Scrape.do requires the target URL to be encoded.
            "url": quote(url, safe=""),
        }
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            print(f"[ScrapedoReader] request failed for {url}: {exc}")
            return None

        # Scrape.do only charges for successful (2xx) responses.
        if response.status_code != 200:
            print(
                f"[ScrapedoReader] {url} returned {response.status_code}: "
                f"{response.text[:200]}"
            )
            return None

        return response.text

    def load_data(
        self,
        urls: List[str],
        render: bool = False,
        super_proxy: bool = False,
    ) -> List[Document]:
        """Scrape each URL and return it as a LlamaIndex Document.

        Failed URLs are skipped, not raised, so one bad page does not abort
        a batch.
        """
        documents: List[Document] = []
        for url in urls:
            content = self._scrape(url, render=render, super_proxy=super_proxy)
            if content is None:
                continue
            documents.append(
                Document(
                    text=content,
                    metadata={"source": url, "scraper": "scrape.do"},
                )
            )
        return documents

Three choices in there matter. The target URL is encoded with quote(url, safe="") because Scrape.do expects an encoded target, skip it and any URL carrying its own query string breaks in ways that look like a site problem. Failures are skipped, not raised, so one dead link in a batch doesn't abort the whole index build; the reader logs it and moves on, and since failed requests aren't billed, those skips are free. And render and super_proxy are per-call flags defaulting off, flip render on for JavaScript-heavy pages and super_proxy on for hard targets or after a ROTATION_FAILED.

That's the entire integration. You can stop here and use the reader on its own:

from scrapedo_llamaindex.reader import ScrapedoReader
from llama_index.core import VectorStoreIndex

docs = ScrapedoReader().load_data(["https://example.com/"])
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What is this page about?"))

Why wrap it in a Workflow

You could keep going in a straight script, scrape, then index, then query, and for a one-off it would be fine. But a real scrape-and-ask pipeline has three stages with genuinely different characters: scraping is slow and fails in messy network-shaped ways, indexing is CPU and embedding work, querying is a fast lookup. Mashing them into one function means the expensive, flaky stage is tangled up with the cheap, reliable ones, and testing any part in isolation gets awkward.

LlamaIndex Workflows became the recommended way to build non-trivial apps precisely because they untangle this. An application is a set of @step methods, each receiving a typed Event, doing one job, and emitting another. The runtime routes each event to whichever step's signature accepts it, so the event types are the edges of your pipeline and plain Python is the logic inside each edge. For scrape-index-query that mapping is exact: three steps, three event types, the slow network stage cleanly walled off from everything downstream.

The Workflow

Create scrapedo_llamaindex/workflow.py. First the events that carry data between steps:

from typing import List

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.schema import Document
from llama_index.core.workflow import (
    Context,
    Event,
    StartEvent,
    StopEvent,
    Workflow,
    step,
)

from scrapedo_llamaindex.reader import ScrapedoReader


class ScrapedEvent(Event):
    """Carries the raw scraped pages from the scrape step to the index step."""

    documents: List[Document]


class IndexedEvent(Event):
    """Carries the built index from the index step to the query step."""

    index: VectorStoreIndex

Then the workflow, three steps, each consuming one event type and emitting the next:

class ScrapedoRAGWorkflow(Workflow):
    """Scrape a set of URLs with Scrape.do, index them, and answer a question."""

    @step
    async def scrape(self, ctx: Context, ev: StartEvent) -> ScrapedEvent:
        # Inputs passed to .run() arrive on the StartEvent.
        urls = ev.urls
        render = getattr(ev, "render", False)
        super_proxy = getattr(ev, "super_proxy", False)

        # Stash the query for the final step.
        await ctx.store.set("query", ev.query)

        reader = ScrapedoReader()
        documents = reader.load_data(
            urls, render=render, super_proxy=super_proxy
        )
        if not documents:
            raise RuntimeError(
                "Scrape.do returned no usable content for any URL. "
                "Try render=True for JS pages or super_proxy=True for hard targets."
            )
        return ScrapedEvent(documents=documents)

    @step
    async def index(self, ctx: Context, ev: ScrapedEvent) -> IndexedEvent:
        splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=100)
        index = VectorStoreIndex.from_documents(
            ev.documents, transformations=[splitter]
        )
        return IndexedEvent(index=index)

    @step
    async def query(self, ctx: Context, ev: IndexedEvent) -> StopEvent:
        query = await ctx.store.get("query")
        query_engine = ev.index.as_query_engine()
        response = await query_engine.aquery(query)
        return StopEvent(result=str(response))

Read the method signatures top to bottom and the whole pipeline is visible without any orchestration code: scrape takes the StartEvent so it runs first, calls the reader, and emits a ScrapedEvent carrying the documents; index takes that ScrapedEvent, splits and embeds the pages, and emits an IndexedEvent; query takes the IndexedEvent, runs the question, and returns a StopEvent that ends the workflow. The runtime wires the steps together by matching types, there's nothing to maintain in between.

One detail worth copying: the query rides in the Context store (ctx.store.set / ctx.store.get) rather than being threaded through every event. Context is for the values a later step needs but the intermediate events shouldn't have to carry.

Running it

Create main.py:

import asyncio

from dotenv import load_dotenv

from scrapedo_llamaindex.workflow import ScrapedoRAGWorkflow

load_dotenv()

URLS = ["https://docs.scrape.do/"]
QUESTION = "What does this service do and what problems does it solve?"


async def run():
    workflow = ScrapedoRAGWorkflow(timeout=120, verbose=True)
    result = await workflow.run(
        urls=URLS,
        query=QUESTION,
        # render=True,       # turn on for JavaScript-heavy pages
        # super_proxy=True,  # turn on for hard targets / ROTATION_FAILED
    )
    print(result)


if __name__ == "__main__":
    asyncio.run(run())

Keys in .env:

SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key

Then:

python main.py

With verbose=True you'll watch each step fire, scrape, index, query, as the events flow through, and the answer comes back grounded in the page that was actually scraped rather than whatever the model happened to remember.

Scaling up

The reader already loops over a list, so bulk ingestion is just a longer URLS. Because only successful responses are billed and the reader skips failures, a batch with a few unreachable pages costs you only the ones you got. When scrape latency starts to dominate a large batch, LlamaIndex Workflows support parallel step execution via @step(num_workers=N), you can fan the scrape stage out across URLs and collect the results with ctx.collect_events(). That's the natural next iteration once the sequential version is solid.

Gotchas

Encode the target URL. quote(url, safe="") is not optional, targets with query strings fail confusingly without it.

A block page is silent. This is the whole reason to route through Scrape.do: a failed scrape doesn't announce itself, it becomes a convincing document in your index. If answers seem off, inspect what actually got scraped before you go blaming retrieval or the prompt.

Escalate deliberately. render=true and super=true both cost more credits. Leave them off and enable them only when a page needs a browser or a residential IP.

Keep the token out of your code. Read SCRAPEDO_TOKEN from the environment. Never commit it or paste it into a screenshot.

Where to take it next

Persist the index. Swap the in-memory VectorStoreIndex for a real vector store so you scrape once and query many times.
Parallelize the scrape step with @step(num_workers=N) across a large URL list.
Add a chat loop. Reuse the same Context across runs so a user keeps asking about the scraped pages without re-scraping.

Wrapping up

LlamaIndex Workflows give you a clean, typed, event-driven skeleton for multi-step LLM apps. What they can't do on their own is keep the open web's block pages and empty shells out of the index they build on. Route the scrape step through Scrape.do and that skeleton takes you from a list of URLs, however well-defended, to grounded answers about their real content, without you managing a single proxy or browser driver.

Get a free Scrape.do token, 1,000 requests a month, no credit card, and point your next LlamaIndex pipeline at the live web.

Downloads: