# Haystack - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/haystack/

Step-by-step guide to integrate Scrape.do with Haystack for LLM pipelines and AI agents

Scrape.do integrates with [Haystack](https://haystack.deepset.ai/), the open-source Python framework by deepset for building LLM applications. The `scrapedo-haystack` package adds a `ScrapeDoFetcher` component that fetches web pages through Scrape.do and returns them as Haystack Documents, ready to pass to an LLM, a RAG pipeline, or a document store.

- Package: [scrapedo-haystack on PyPI](https://pypi.org/project/scrapedo-haystack/)
- Source code: [GitHub](https://github.com/bugrahansss/scrapedo-haystack)

## 1. Install the Package and Set Your Token

Let's start by installing the package. It needs Python 3.10 or later.

```bash
pip install scrapedo-haystack
```

This also installs Haystack (`haystack-ai`) if you don't have it yet.

Next, set your Scrape.do API token as an environment variable. You can find your token in the [Scrape.do dashboard](https://dashboard.scrape.do/).

```bash
export SCRAPEDO_API_KEY="your-token"
```

`ScrapeDoFetcher` reads the token from `SCRAPEDO_API_KEY` automatically, so you don't need to put it in your code.

## 2. Send a Request to Scrape.do

Create a fetcher and pass it a list of URLs. Every URL that is fetched successfully becomes one Document.

```python
from scrapedo_haystack import ScrapeDoFetcher

fetcher = ScrapeDoFetcher()
result = fetcher.run(urls=["https://example.com"])

doc = result["documents"][0]
print(doc.meta["url"])          # the requested URL
print(doc.meta["status_code"])  # HTTP status returned by Scrape.do
print(doc.content)              # page metadata, then the page as Markdown
```

Pages are returned as Markdown by default, which is the easiest format for an LLM to read. Set `output="raw"` to get the original HTML instead.

The fetcher supports the most common Scrape.do parameters as arguments:

```python
fetcher = ScrapeDoFetcher(
    output="markdown",   # "markdown" or "raw" (original HTML)
    render=True,         # run JavaScript in a headless browser
    super_proxy=True,    # use residential and mobile IPs
    geo_code="us",       # route the request through a country
    timeout=60,          # request timeout in seconds
    max_workers=4,       # URLs fetched in parallel
)
```

| Argument | Scrape.do parameter | When to use it |
|---|---|---|
| `render=True` | `render` | The page loads its content with JavaScript |
| `super_proxy=True` | `super` | The site blocks datacenter IPs |
| `geo_code="us"` | `geoCode` | You need the page as seen from a specific country |
| `output="raw"` | `output` | You want HTML instead of Markdown |

Any other Scrape.do parameter can be passed with `extra_params`:

```python
fetcher = ScrapeDoFetcher(render=True, extra_params={"waitUntil": "networkidle0"})
```

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. Then set the same options on `ScrapeDoFetcher`: **Super** is `super_proxy=True`, **Render JavaScript** is `render=True`, and the rest go in `extra_params`, for example `{"blockResources": "false"}`.

By default, a URL that fails is logged and skipped, and the other URLs are still returned. Set `raise_on_failure=True` to stop with an error instead. If you get an empty list back, check that `SCRAPEDO_API_KEY` is set: without a token, every URL fails.

## 3. Use It in a Pipeline

Now let's connect the fetcher to an LLM. The pipeline below fetches a page and answers a question using only that page's content.

This example uses OpenAI, so it also needs `OPENAI_API_KEY` set. Any Haystack chat generator works the same way.

```python
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from scrapedo_haystack import ScrapeDoFetcher

template = [
    ChatMessage.from_user(
        """Answer the question using only these pages.

{% for doc in documents %}
Source: {{ doc.meta.url }}
{{ doc.content }}
{% endfor %}

Question: {{ question }}"""
    )
]

pipe = Pipeline()
pipe.add_component("fetcher", ScrapeDoFetcher(render=True))
pipe.add_component("prompt", ChatPromptBuilder(template=template, required_variables=["question"]))
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))

pipe.connect("fetcher.documents", "prompt.documents")
pipe.connect("prompt.prompt", "llm.messages")

result = pipe.run({
    "fetcher": {"urls": ["https://haystack.deepset.ai/overview/intro"]},
    "prompt": {"question": "What is Haystack?"},
})
print(result["llm"]["replies"][0].text)
```

To store pages for search instead, connect `fetcher.documents` to a document writer or an embedder. The fetcher already returns Documents, so you don't need a converter.

## 4. Save and Load the Pipeline

Haystack pipelines can be saved as YAML and loaded later. Your API token is never written to the file; it is read from `SCRAPEDO_API_KEY` when the pipeline runs.

Haystack 3 only loads components from modules you allow, so pass this package's module when loading:

```python
yaml_text = pipe.dumps()

pipe = Pipeline.loads(yaml_text, allowed_modules=["scrapedo_haystack.fetcher"])
```

On Haystack 2, call `Pipeline.loads(yaml_text)` without `allowed_modules`. Haystack 2 has no such argument and raises a `TypeError` if you pass it.