# Airflow - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/airflow/

Step-by-step guide to integrate Scrape.do with Apache Airflow for scheduled web scraping pipelines

Scrape.do works in Apache Airflow as a plain HTTP request inside a DAG. The template below runs a scrape every day and retries automatically when a request fails.

## 1. Install and Start Airflow

In a terminal, create a workspace folder, a virtual environment and the Airflow install:

```bash
mkdir scrape_do_airflow && cd scrape_do_airflow
python3 -m venv venv && source venv/bin/activate
pip install apache-airflow requests
airflow standalone
```

Airflow prints a one-time `admin` password to the console while it starts. Copy it.

## 2. Log In to the Dashboard

Open `localhost:8080` in your browser and log in with username `admin` and the password you copied.

## 3. Add the Scrape.do DAG

`airflow standalone` creates an `airflow/` folder in your home directory (`~/airflow`, or wherever `AIRFLOW_HOME` points). Inside it, create a `dags/` folder and save this file as `scrape_do_template.py`:

```python
from datetime import datetime, timedelta

import requests
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator

def scrape_do_task(**kwargs):
    token = "<your_token>"  # Your Scrape.do API token
    target_url = "https://books.toscrape.com/"

    response = requests.get(
        "https://api.scrape.do/",
        params={"token": token, "url": target_url, "super": "true", "render": "true"},
        timeout=45,
    )
    if response.status_code != 200:
        # Raising makes Airflow retry the task
        raise ValueError(f"Error Code: {response.status_code}")

    print(response.text[:200])
    return "Scraping Completed"

default_args = {
    "owner": "scrape_do",
    "depends_on_past": False,
    "retries": 2,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="scrape_do_universal_pipeline",
    default_args=default_args,
    description="Scheduled scraping template powered by Scrape.do",
    start_date=datetime(2026, 5, 1),
    schedule="@daily",
    catchup=False,
    tags=["scrape.do"],
) as dag:
    run_scraper = PythonOperator(
        task_id="execute_scrape_do_request",
        python_callable=scrape_do_task,
    )
```

- Replace `<your_token>` with your API token from your [dashboard](https://dashboard.scrape.do) and `target_url` with the page you want.
- `super=true` uses residential and mobile proxies and `render=true` renders the page in a headless browser. Drop either one for targets that don't need it, since both add credits. See [request costs](/documentation/request-costs).
- `requests` encodes the target URL in `params` for you.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response.

## 4. Run the Pipeline

The DAG appears in the dashboard as `scrape_do_universal_pipeline` without a restart.

- Toggle it **ON** with the switch on the DAG row.
- Click the **Play** button to trigger a run now.
- Open the task instance and switch to the **Logs** tab to watch the request.

From then on it runs once a day. A failed run waits 2 minutes and retries, and the task only fails after the third attempt.

Full tutorial with every step: [Web Scraper for Airflow: The Ultimate Scheduled Pipeline Template (Powered by Scrape.do)](/blog/airflow-web-scraper/)