Haystack - Scrape.do Integration

Scrape.do integrates with Haystack, the open-source Python framework by deepset for building LLM applications. The scrapedo-haystack package adds a ScrapeDoFetcher component that fetches web pages through Scrape.do and returns them as Haystack Documents, ready to pass to an LLM, a RAG pipeline, or a document store.

• Package: scrapedo-haystack on PyPI
• Source code: GitHub

1. Install the Package and Set Your Token

Let's start by installing the package. It needs Python 3.10 or later.

pip install scrapedo-haystack

This also installs Haystack (haystack-ai) if you don't have it yet.

Next, set your Scrape.do API token as an environment variable. You can find your token in the Scrape.do dashboard.

export SCRAPEDO_API_KEY="your-token"

ScrapeDoFetcher reads the token from SCRAPEDO_API_KEY automatically, so you don't need to put it in your code.

2. Send a Request to Scrape.do

Create a fetcher and pass it a list of URLs. Every URL that is fetched successfully becomes one Document.

from scrapedo_haystack import ScrapeDoFetcher

fetcher = ScrapeDoFetcher()
result = fetcher.run(urls=["https://example.com"])

doc = result["documents"][0]
print(doc.meta["url"])          # the requested URL
print(doc.meta["status_code"])  # HTTP status returned by Scrape.do
print(doc.content)              # page metadata, then the page as Markdown

Pages are returned as Markdown by default, which is the easiest format for an LLM to read. Set output="raw" to get the original HTML instead.

The fetcher supports the most common Scrape.do parameters as arguments:

fetcher = ScrapeDoFetcher(
    output="markdown",   # "markdown" or "raw" (original HTML)
    render=True,         # run JavaScript in a headless browser
    super_proxy=True,    # use residential and mobile IPs
    geo_code="us",       # route the request through a country
    timeout=60,          # request timeout in seconds
    max_workers=4,       # URLs fetched in parallel
)

Any other Scrape.do parameter can be passed with extra_params:

fetcher = ScrapeDoFetcher(render=True, extra_params={"waitUntil": "networkidle0"})

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Then set the same options on ScrapeDoFetcher: Super is super_proxy=True, Render JavaScript is render=True, and the rest go in extra_params, for example {"blockResources": "false"}.

By default, a URL that fails is logged and skipped, and the other URLs are still returned. Set raise_on_failure=True to stop with an error instead. If you get an empty list back, check that SCRAPEDO_API_KEY is set: without a token, every URL fails.

3. Use It in a Pipeline

Now let's connect the fetcher to an LLM. The pipeline below fetches a page and answers a question using only that page's content.

This example uses OpenAI, so it also needs OPENAI_API_KEY set. Any Haystack chat generator works the same way.

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from scrapedo_haystack import ScrapeDoFetcher

template = [
    ChatMessage.from_user(
        """Answer the question using only these pages.

{% for doc in documents %}
Source: {{ doc.meta.url }}
{{ doc.content }}
{% endfor %}

Question: {{ question }}"""
    )
]

pipe = Pipeline()
pipe.add_component("fetcher", ScrapeDoFetcher(render=True))
pipe.add_component("prompt", ChatPromptBuilder(template=template, required_variables=["question"]))
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))

pipe.connect("fetcher.documents", "prompt.documents")
pipe.connect("prompt.prompt", "llm.messages")

result = pipe.run({
    "fetcher": {"urls": ["https://haystack.deepset.ai/overview/intro"]},
    "prompt": {"question": "What is Haystack?"},
})
print(result["llm"]["replies"][0].text)

To store pages for search instead, connect fetcher.documents to a document writer or an embedder. The fetcher already returns Documents, so you don't need a converter.

4. Save and Load the Pipeline

Haystack pipelines can be saved as YAML and loaded later. Your API token is never written to the file; it is read from SCRAPEDO_API_KEY when the pipeline runs.

Haystack 3 only loads components from modules you allow, so pass this package's module when loading:

yaml_text = pipe.dumps()

pipe = Pipeline.loads(yaml_text, allowed_modules=["scrapedo_haystack.fetcher"])

On Haystack 2, call Pipeline.loads(yaml_text) without allowed_modules. Haystack 2 has no such argument and raises a TypeError if you pass it.