# LlamaIndex - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/llamaindex/

Step-by-step guide to integrate Scrape.do with LlamaIndex to index and query live web pages

Scrape.do integrates with LlamaIndex through a small reader, `ScrapedoReader`, that loads any URL as a LlamaIndex `Document`. Scrape.do handles proxy rotation, anti-bot bypass, CAPTCHAs and JavaScript rendering, so the pages that enter your index are the real pages, not block screens.

## 1. Install the Packages and Set Your Keys

You need Python 3.10 or later:

```bash
pip install llama-index llama-index-llms-openai requests python-dotenv
```

Create a `.env` file with your Scrape.do token from your [dashboard](https://dashboard.scrape.do) and an OpenAI key (used for embeddings and answers):

```
SCRAPEDO_TOKEN=your_scrapedo_token
OPENAI_API_KEY=your_openai_key
```

## 2. Add the Reader

Create `scrapedo_llamaindex/reader.py`:

```python
import os
from typing import List, Optional

import requests
from llama_index.core.schema import Document

SCRAPEDO_ENDPOINT = "https://api.scrape.do/"

class ScrapedoReader:
    """Load web pages as LlamaIndex Documents via the Scrape.do API."""

    def __init__(self, api_token: Optional[str] = None, timeout: int = 60):
        self.api_token = api_token or os.getenv("SCRAPEDO_TOKEN")
        if not self.api_token:
            raise ValueError(
                "Scrape.do token missing. Set the SCRAPEDO_TOKEN environment "
                "variable or pass api_token=... when creating the reader."
            )
        self.timeout = timeout

    def _scrape(self, url: str, render: bool, super_proxy: bool) -> Optional[str]:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params = {"token": self.api_token, "url": url}
        if render:
            params["render"] = "true"
        if super_proxy:
            params["super"] = "true"

        try:
            response = requests.get(
                SCRAPEDO_ENDPOINT, params=params, timeout=self.timeout
            )
        except requests.RequestException as exc:
            print(f"[ScrapedoReader] request failed for {url}: {exc}")
            return None

        # Failed requests are not charged; see Request Costs in the docs for which statuses are.
        if response.status_code != 200:
            print(
                f"[ScrapedoReader] {url} returned {response.status_code}: "
                f"{response.text[:200]}"
            )
            return None

        return response.text

    def load_data(
        self,
        urls: List[str],
        render: bool = False,
        super_proxy: bool = False,
    ) -> List[Document]:
        """Scrape each URL and return it as a LlamaIndex Document.

        Failed URLs are skipped, not raised, so one bad page does not abort
        a batch.
        """
        documents: List[Document] = []
        for url in urls:
            content = self._scrape(url, render=render, super_proxy=super_proxy)
            if content is None:
                continue
            documents.append(
                Document(
                    text=content,
                    metadata={"source": url, "scraper": "scrape.do"},
                )
            )
        return documents
```

Pass the target URL as it is. `requests` encodes it once, and encoding it yourself as well makes Scrape.do reject the request with `Your target 'URL' is not valid!`.

## 3. Index a Page and Ask About It

```python
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex

from scrapedo_llamaindex.reader import ScrapedoReader

load_dotenv()

docs = ScrapedoReader().load_data(["https://example.com/"])
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What is this page about?"))
```

## 4. Turn On Rendering or Residential Proxies When Needed

Both flags default to off because they cost more credits:

```python
docs = ScrapedoReader().load_data(urls, render=True)       # JavaScript-heavy pages
docs = ScrapedoReader().load_data(urls, super_proxy=True)  # hard targets or ROTATION_FAILED
```

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. Then pass the same options to `load_data()`: **Super** is `super_proxy=True` and **Render JavaScript** is `render=True`.

The tutorial also wraps the reader in an event-driven LlamaIndex Workflow that scrapes, indexes and answers in three typed steps.

Full tutorial with every step: [Web Scraping in LlamaIndex: A Workflow That Reads the Live Web with Scrape.do](/blog/llamaindex-web-scraping/)