LangChain - Scrape.do Integration | Documentation

Step-by-step guide to integrate Scrape.do with LangChain document loaders, RAG pipelines and agents

Scrape.do integrates with LangChain through a custom document loader, ScrapedoLoader. It fetches each URL through Scrape.do, which handles proxy rotation, anti-bot bypass and JavaScript rendering, and returns standard LangChain Document objects, so your splitting, embedding and retrieval stay the same.

Download scrapedo_loader.py to drop the loader into your project, then replace YOUR_SCRAPEDO_TOKEN with your token.

1. Install the Packages

You need Python 3.10 or later:

pip install langchain-core langchain-text-splitters requests

Then copy your API token from your Scrape.do dashboard.

2. Add the Loader

Save this as scrapedo_loader.py. It implements LangChain's BaseLoader, so it works anywhere a document loader does. .load() returns every document at once and .lazy_load() streams them one at a time.

from typing import Any, Dict, Iterator, List, Optional, Union

import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

SCRAPEDO_API_URL = "https://api.scrape.do/"


class ScrapedoLoader(BaseLoader):
    """Load web pages via the Scrape.do API into LangChain Documents."""

    def __init__(
        self,
        token: str,
        urls: Union[str, List[str]],
        render: bool = False,
        super_proxy: bool = False,
        extra_params: Optional[Dict[str, Any]] = None,
        timeout: int = 90,
        continue_on_failure: bool = True,
    ) -> None:
        if not token:
            raise ValueError("A Scrape.do API token is required.")
        self.token = token
        self.urls = [urls] if isinstance(urls, str) else list(urls)
        self.render = render
        self.super_proxy = super_proxy
        self.extra_params = extra_params or {}
        self.timeout = timeout
        self.continue_on_failure = continue_on_failure

    def _build_params(self, url: str) -> Dict[str, Any]:
        # requests URL-encodes parameter values automatically,
        # which satisfies Scrape.do's encoded-URL requirement.
        params: Dict[str, Any] = {"token": self.token, "url": url}
        if self.render:
            params["render"] = "true"
        if self.super_proxy:
            params["super"] = "true"
        params.update(self.extra_params)
        return params

    def lazy_load(self) -> Iterator[Document]:
        for url in self.urls:
            try:
                response = requests.get(
                    SCRAPEDO_API_URL,
                    params=self._build_params(url),
                    timeout=self.timeout,
                )
                response.raise_for_status()
            except requests.RequestException as exc:
                # Never leak the API token in error output.
                safe_error = str(exc).replace(self.token, "***TOKEN***")
                if self.continue_on_failure:
                    print(f"[ScrapedoLoader] Skipping {url}: {safe_error}")
                    continue
                raise requests.RequestException(safe_error) from None

            yield Document(
                page_content=response.text,
                metadata={
                    "source": url,
                    "status_code": response.status_code,
                    "render": self.render,
                    "super_proxy": self.super_proxy,
                },
            )

    def load(self) -> List[Document]:
        return list(self.lazy_load())

With continue_on_failure=True (the default), a URL that fails is logged and skipped instead of stopping the batch.

3. Load a Page

from scrapedo_loader import ScrapedoLoader

loader = ScrapedoLoader(
    token="YOUR_SCRAPEDO_TOKEN",
    urls=["https://httpbin.co/anything"],
)

docs = loader.load()
print(docs[0].metadata["status_code"])   # 200
print(docs[0].page_content[:300])

For a JavaScript-heavy page, pass render=True. If a site keeps rejecting you, pass super_proxy=True to use residential and mobile proxies. Any other Scrape.do parameter goes in extra_params. Rendered and super requests cost more credits, so turn them on only when a target needs them.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Then set the same options on ScrapedoLoader: Super is super_proxy=True, Render JavaScript is render=True, and the rest go in extra_params.

4. Use It in a RAG Pipeline

The loader returns standard Document objects, so you split and embed them as usual. The splitter below comes from the langchain-text-splitters package.

from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = ScrapedoLoader(
    token="YOUR_SCRAPEDO_TOKEN",
    urls=["https://example.com/docs", "https://example.com/faq"],
    render=True,
)
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)

metadata["source"] keeps the original URL on every chunk, so citations point back to the page they came from. Load your token from an environment variable rather than pasting it into source code.

Full tutorial with every step: Scraping JavaScript-Heavy Pages into LangChain with Scrape.do