# LangChain - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/langchain/

Step-by-step guide to integrate Scrape.do with LangChain document loaders, RAG pipelines and agents

Scrape.do integrates with LangChain through a custom document loader, `ScrapedoLoader`. It fetches each URL through Scrape.do, which handles proxy rotation, anti-bot bypass and JavaScript rendering, and returns standard LangChain `Document` objects, so your splitting, embedding and retrieval stay the same.

**<a href="/uploads/blog/scrapedo_loader.py" download="scrapedo_loader.py">Download scrapedo_loader.py</a>** to drop the loader into your project, then replace `YOUR_SCRAPEDO_TOKEN` with your token.

## 1. Install the Packages

You need Python 3.10 or later:

```bash
pip install langchain-core langchain-text-splitters requests
```

Then copy your API token from your [Scrape.do dashboard](https://dashboard.scrape.do).

## 2. Add the Loader

Save this as `scrapedo_loader.py`. It implements LangChain's `BaseLoader`, so it works anywhere a document loader does. `.load()` returns every document at once and `.lazy_load()` streams them one at a time.

```python
from typing import Any, Dict, Iterator, List, Optional, Union

import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

SCRAPEDO_API_URL = "https://api.scrape.do/"

class ScrapedoLoader(BaseLoader):
    """Load web pages via the Scrape.do API into LangChain Documents."""

    def __init__(
        self,
        token: str,
        urls: Union[str, List[str]],
        render: bool = False,
        super_proxy: bool = False,
        extra_params: Optional[Dict[str, Any]] = None,
        timeout: int = 90,
        continue_on_failure: bool = True,
    ) -> None:
        if not token:
            raise ValueError("A Scrape.do API token is required.")
        self.token = token
        self.urls = [urls] if isinstance(urls, str) else list(urls)
        self.render = render
        self.super_proxy = super_proxy
        self.extra_params = extra_params or {}
        self.timeout = timeout
        self.continue_on_failure = continue_on_failure

    def _build_params(self, url: str) -> Dict[str, Any]:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params: Dict[str, Any] = {"token": self.token, "url": url}
        if self.render:
            params["render"] = "true"
        if self.super_proxy:
            params["super"] = "true"
        params.update(self.extra_params)
        return params

    def lazy_load(self) -> Iterator[Document]:
        for url in self.urls:
            try:
                response = requests.get(
                    SCRAPEDO_API_URL,
                    params=self._build_params(url),
                    timeout=self.timeout,
                )
                response.raise_for_status()
            except requests.RequestException as exc:
                # Never leak the API token in error output.
                safe_error = str(exc).replace(self.token, "***TOKEN***")
                if self.continue_on_failure:
                    print(f"[ScrapedoLoader] Skipping {url}: {safe_error}")
                    continue
                raise requests.RequestException(safe_error) from None

            yield Document(
                page_content=response.text,
                metadata={
                    "source": url,
                    "status_code": response.status_code,
                    "render": self.render,
                    "super_proxy": self.super_proxy,
                },
            )

    def load(self) -> List[Document]:
        return list(self.lazy_load())
```

With `continue_on_failure=True` (the default), a URL that fails is logged and skipped instead of stopping the batch.

## 3. Load a Page

```python
from scrapedo_loader import ScrapedoLoader

loader = ScrapedoLoader(
    token="YOUR_SCRAPEDO_TOKEN",
    urls=["https://httpbin.co/anything"],
)

docs = loader.load()
print(docs[0].metadata["status_code"])   # 200
print(docs[0].page_content[:300])
```

For a JavaScript-heavy page, pass `render=True`. If a site keeps rejecting you, pass `super_proxy=True` to use residential and mobile proxies. Any other Scrape.do parameter goes in `extra_params`. Rendered and super requests cost more credits, so turn them on only when a target needs them.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. Then set the same options on `ScrapedoLoader`: **Super** is `super_proxy=True`, **Render JavaScript** is `render=True`, and the rest go in `extra_params`.

## 4. Use It in a RAG Pipeline

The loader returns standard `Document` objects, so you split and embed them as usual. The splitter below comes from the `langchain-text-splitters` package.

```python
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = ScrapedoLoader(
    token="YOUR_SCRAPEDO_TOKEN",
    urls=["https://example.com/docs", "https://example.com/faq"],
    render=True,
)
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
```

`metadata["source"]` keeps the original URL on every chunk, so citations point back to the page they came from. Load your token from an environment variable rather than pasting it into source code.

Full tutorial with every step: [Scraping JavaScript-Heavy Pages into LangChain with Scrape.do](/blog/scraping-with-langchain/)