LlamaIndex - Scrape.do Integration | Documentation

Step-by-step guide to integrate Scrape.do with LlamaIndex to index and query live web pages

Scrape.do integrates with LlamaIndex through a small reader, ScrapedoReader, that loads any URL as a LlamaIndex Document. Scrape.do handles proxy rotation, anti-bot bypass, CAPTCHAs and JavaScript rendering, so the pages that enter your index are the real pages, not block screens.

1. Install the Packages and Set Your Keys

You need Python 3.10 or later:

pip install llama-index llama-index-llms-openai requests python-dotenv

Create a .env file with your Scrape.do token from your dashboard and an OpenAI key (used for embeddings and answers):

SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key

2. Add the Reader

Create scrapedo_llamaindex/reader.py:

import os
from typing import List, Optional

import requests
from llama_index.core.schema import Document

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"


class ScrapedoReader:
    """Load web pages as LlamaIndex Documents via the Scrape.do API."""

    def __init__(self, api_token: Optional[str] = None, timeout: int = 60):
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the reader."
            )
        self.timeout = timeout

    def _scrape(self, url: str, render: bool, super_proxy: bool) -> Optional[str]:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params = {"token": self.api_token, "url": url}
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            print(f"[ScrapedoReader] request failed for {url}: {exc}")
            return None

        # Failed requests are not charged; see Request Costs in the docs for which statuses are.
        if response.status_code != 200:
            print(
                f"[ScrapedoReader] {url} returned {response.status_code}: "
                f"{response.text[:200]}"
            )
            return None

        return response.text

    def load_data(
        self,
        urls: List[str],
        render: bool = False,
        super_proxy: bool = False,
    ) -> List[Document]:
        """Scrape each URL and return it as a LlamaIndex Document.

        Failed URLs are skipped, not raised, so one bad page does not abort
        a batch.
        """
        documents: List[Document] = []
        for url in urls:
            content = self._scrape(url, render=render, super_proxy=super_proxy)
            if content is None:
                continue
            documents.append(
                Document(
                    text=content,
                    metadata={"source": url, "scraper": "scrape.do"},
                )
            )
        return documents

Pass the target URL as it is. requests encodes it once, and encoding it yourself as well makes Scrape.do reject the request with Your target 'URL' is not valid!.

3. Index a Page and Ask About It

from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex

from scrapedo_llamaindex.reader import ScrapedoReader

load_dotenv()

docs = ScrapedoReader().load_data(["https://example.com/"])
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What is this page about?"))

4. Turn On Rendering or Residential Proxies When Needed

Both flags default to off because they cost more credits:

docs = ScrapedoReader().load_data(urls, render=True)       # JavaScript-heavy pages
docs = ScrapedoReader().load_data(urls, super_proxy=True)  # hard targets or ROTATION_FAILED

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Then pass the same options to load_data(): Super is super_proxy=True and Render JavaScript is render=True.

The tutorial also wraps the reader in an event-driven LlamaIndex Workflow that scrapes, indexes and answers in three typed steps.

Full tutorial with every step: Web Scraping in LlamaIndex: A Workflow That Reads the Live Web with Scrape.do