Categories:Scraping Use Cases,Scraping Tools
Scraping JavaScript-Heavy Pages into LangChain with Scrape.do
Growth
Point LangChain's WebBaseLoader at a modern website and you'll often get a Document back that's technically valid and practically useless. The page rendered its real content with JavaScript that a plain HTTP request never runs, or Cloudflare served a challenge, or the datacenter IP got geo-blocked before the loader saw a single byte. The loader isn't broken. The open web just fights back, and requests.get() under the hood loses that fight on exactly the pages you most want in your index.
This post fixes that by wiring a custom LangChain Document Loader to Scrape.do, a web scraping API that deals with proxy rotation, anti-bot bypass, CAPTCHA solving, and JavaScript rendering on the server side. Your code sends a URL and gets back the fully rendered page. Everything downstream, the splitting, embedding, and retrieval, stays exactly as you already have it. If you've done any web scraping in Python before, the loader will look familiar; the difference is that this one actually gets through.
By the end you'll have three things: a reusable ScrapedoLoader, a bulk script that turns a Google Sheet of URLs into a scraped dataset, and a small agent tool for live fetching. The Python files for the first two are attached at the bottom of the post, ready to drop into your project.
Why Scrape.do Instead of Plain Requests
A custom loader that just wraps requests.get() fails on the same pages WebBaseLoader already fails on, because it's the same request underneath. The point of routing through Scrape.do is that the hard parts move to the API layer, where they belong. It rotates proxies for you (datacenter by default, residential and mobile with super=true when a target is stubborn), handles anti-bot systems and CAPTCHAs invisibly, and renders JavaScript in a real headless browser when you pass render=true.
Two things matter while you're building. You're only billed for successful 2xx responses, so the dead URLs and transient blocks you'll inevitably hit during testing cost nothing. And the free tier gives you 1,000 requests a month, which is enough to write and validate everything here before you decide to pay for anything. Grab a token on the free tier and follow along.
Setup
You need Python 3.10+ and two packages:
pip install langchain-core requestsThen sign up at scrape.do, copy the token from your dashboard, and keep it handy for the code below.
The ScrapedoLoader
Save the following as scrapedo_loader.py. It implements LangChain's BaseLoader interface, so it slots into any pipeline that accepts a document loader. .load() returns everything at once; .lazy_load() streams documents one at a time, which is what you want when the URL list is long.
"""Scrape.do integration for LangChain."""
from typing import Any, Dict, Iterator, List, Optional, Union
import requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document
SCRAPEDO_API_URL = "https://api.scrape.do/"
class ScrapedoLoader(BaseLoader):
"""Load web pages via the Scrape.do API into LangChain Documents."""
def __init__(
self,
token: str,
urls: Union[str, List[str]],
render: bool = False,
super_proxy: bool = False,
extra_params: Optional[Dict[str, Any]] = None,
timeout: int = 90,
continue_on_failure: bool = True,
) -> None:
if not token:
raise ValueError("A Scrape.do API token is required.")
self.token = token
self.urls = [urls] if isinstance(urls, str) else list(urls)
self.render = render
self.super_proxy = super_proxy
self.extra_params = extra_params or {}
self.timeout = timeout
self.continue_on_failure = continue_on_failure
def _build_params(self, url: str) -> Dict[str, Any]:
# requests URL-encodes parameter values automatically,
# which satisfies Scrape.do's encoded-URL requirement.
params: Dict[str, Any] = {"token": self.token, "url": url}
if self.render:
params["render"] = "true"
if self.super_proxy:
params["super"] = "true"
params.update(self.extra_params)
return params
def lazy_load(self) -> Iterator[Document]:
for url in self.urls:
try:
response = requests.get(
SCRAPEDO_API_URL,
params=self._build_params(url),
timeout=self.timeout,
)
response.raise_for_status()
except requests.RequestException as exc:
# Never leak the API token in error output.
safe_error = str(exc).replace(self.token, "***TOKEN***")
if self.continue_on_failure:
print(f"[ScrapedoLoader] Skipping {url}: {safe_error}")
continue
raise requests.RequestException(safe_error) from None
yield Document(
page_content=response.text,
metadata={
"source": url,
"status_code": response.status_code,
"render": self.render,
"super_proxy": self.super_proxy,
},
)
def load(self) -> List[Document]:
return list(self.lazy_load())Three details in there are worth calling out. Scrape.do wants the target URL URL-encoded, and passing it through the params argument of requests handles that for you, so there are no manual urllib.parse.quote calls to get wrong. With continue_on_failure=True (the default), a dead URL gets logged and skipped rather than killing a batch halfway through, and because you're only charged for successful responses, those skips don't show up on your bill. Finally, the error string is scrubbed before it's printed, so a live token never ends up pasted into a GitHub issue or a screenshot.
Test It
from scrapedo_loader import ScrapedoLoader
loader = ScrapedoLoader(
token="YOUR_SCRAPEDO_TOKEN",
urls=["https://httpbin.co/anything"],
)
docs = loader.load()
print(docs[0].metadata["status_code"]) # 200
print(docs[0].page_content[:300])For a JavaScript-heavy page, flip one switch:
loader = ScrapedoLoader(
token="YOUR_SCRAPEDO_TOKEN",
urls=["https://example.com/spa-page"],
render=True, # full headless-browser rendering
)Rendered requests spin up a real browser and cost more credits than a plain fetch, so reach for render=True when a target actually needs it rather than setting it by default.
Bulk Scraping From a Google Sheet
One URL is a demo. A hundred URLs is a workflow. The script below reads a list of URLs straight out of a Google Sheet, runs them all through ScrapedoLoader, and writes the results to a CSV with the URL, status, and full page content on each row.
First, prepare the sheet. Put your URLs in a column headed url, set sharing to Anyone with the link – Viewer, and build the CSV export link from the sheet's ID:
https://docs.google.com/spreadsheets/d/SHEET_ID/export?format=csv
There's no OAuth, no Google Cloud project, and no service account involved. The export link is just a URL that requests can fetch.
Save this as bulk_scrape.py in the same folder as scrapedo_loader.py:
"""Bulk web scraping with Scrape.do + LangChain."""
import csv
import io
import sys
import requests
from scrapedo_loader import ScrapedoLoader
# ----------------------- CONFIG -----------------------
SCRAPEDO_TOKEN = "YOUR_SCRAPEDO_TOKEN"
SHEET_CSV_URL = "YOUR_SHEET_CSV_LINK"
URL_COLUMN = "url"
RENDER = False
OUTPUT_FILE = "results.csv"
# -------------------------------------------------------
def read_urls_from_sheet(sheet_csv_url: str, url_column: str) -> list:
response = requests.get(sheet_csv_url, timeout=30)
response.raise_for_status()
reader = csv.DictReader(io.StringIO(response.text))
if reader.fieldnames is None or url_column not in reader.fieldnames:
sys.exit(
f"Column '{url_column}' not found in the sheet. "
f"Available columns: {reader.fieldnames}"
)
return [row[url_column].strip() for row in reader if row.get(url_column, "").strip()]
def main() -> None:
urls = read_urls_from_sheet(SHEET_CSV_URL, URL_COLUMN)
if not urls:
sys.exit("No URLs found in the sheet.")
print(f"Found {len(urls)} URLs. Scraping...")
loader = ScrapedoLoader(
token=SCRAPEDO_TOKEN,
urls=urls,
render=RENDER,
continue_on_failure=True,
)
scraped = {}
for doc in loader.lazy_load():
scraped[doc.metadata["source"]] = doc
print(f" OK {doc.metadata['source']}")
with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["url", "status", "content"])
for url in urls:
doc = scraped.get(url)
if doc:
writer.writerow([url, doc.metadata["status_code"], doc.page_content])
else:
writer.writerow([url, "FAILED", ""])
ok = len(scraped)
print(f"\nDone. {ok} succeeded, {len(urls) - ok} failed.")
print(f"Results saved to {OUTPUT_FILE}")
if __name__ == "__main__":
main()Fill in the two config values, then run:
python3 bulk_scrape.pyEvery row in your sheet becomes a row in results.csv. A URL that fails gets marked FAILED instead of crashing the whole run, and since those failures don't return a 2xx, they don't cost you a credit either.
Plugging It Into a RAG Pipeline
Because ScrapedoLoader returns standard LangChain Document objects, the rest of your RAG stack doesn't care where they came from. Scrape, split, embed, store:
from langchain_text_splitters import RecursiveCharacterTextSplitter
loader = ScrapedoLoader(
token="YOUR_SCRAPEDO_TOKEN",
urls=["https://example.com/docs", "https://example.com/faq"],
render=True,
)
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
# From here it's your usual pipeline:
# embeddings -> vector store -> retriever -> chainThe metadata["source"] field carries the original URL through every stage, so when your retriever cites a chunk, the citation points back to the page it actually came from without any extra bookkeeping on your side.
A Scrape.do Tool for Agents
Loaders are for pipelines; tools are for agents. This small factory wraps the loader into a LangChain tool that an agent can call whenever it decides it needs live web content:
def make_scrapedo_tool(token: str, render: bool = False):
from langchain_core.tools import tool
@tool
def scrapedo_scrape(url: str) -> str:
"""Fetch the full content of a web page. Handles JavaScript
rendering, proxies, and anti-bot protection. Input: a URL."""
loader = ScrapedoLoader(token=token, urls=url, render=render)
docs = loader.load()
if not docs:
return f"Failed to fetch {url}."
return docs[0].page_content
return scrapedo_scrapePass make_scrapedo_tool(token="...") into your agent's tool list and it picks up a dependable "go read this page" ability. No more agents throwing up their hands on the first 403 and hallucinating the rest.
Gotchas Worth Knowing
render=True costs more credits, so start without it and only enable it when a plain response comes back incomplete. If a site keeps rejecting you even then, super_proxy=True routes the request through residential and mobile proxies, which usually get through where datacenter IPs don't. Raw HTML is noisy for embeddings; running the content through BeautifulSoup's get_text() before you split it gives cleaner chunks and better retrieval. And keep your token out of anything that leaves your machine by loading it from an environment variable rather than pasting it into the source.
Wrapping Up
In about eighty lines of Python, "WebBaseLoader returned an empty page again" turns into a loader that renders JavaScript, rotates proxies, and gets past anti-bot protection, plus a bulk workflow and an agent tool built on the same foundation. The next step is your own data: swap the demo URLs for the sites your pipeline actually needs, and the same loader handles the ones that used to come back blank. If you're new to the underlying techniques, the Python web scraping guide is a good companion read.
Downloads:
scrapedo_loader.py— the LangChain document loaderbulk_scrape.py— the bulk scraping script
Scrape.do's free tier includes 1,000 requests per month and only bills you for successful responses. Get your free API token and point your RAG pipeline at the real web.
Growth
