Airflow - Scrape.do Integration

Scrape.do works in Apache Airflow as a plain HTTP request inside a DAG. The template below runs a scrape every day and retries automatically when a request fails.

1. Install and Start Airflow

In a terminal, create a workspace folder, a virtual environment and the Airflow install:

mkdir scrape_do_airflow && cd scrape_do_airflow
python3 -m venv venv && source venv/bin/activate
pip install apache-airflow requests
airflow standalone

Airflow prints a one-time admin password to the console while it starts. Copy it.

2. Log In to the Dashboard

Open localhost:8080 in your browser and log in with username admin and the password you copied.

3. Add the Scrape.do DAG

airflow standalone creates an airflow/ folder in your home directory (~/airflow, or wherever AIRFLOW_HOME points). Inside it, create a dags/ folder and save this file as scrape_do_template.py:

from datetime import datetime, timedelta

import requests
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator


def scrape_do_task(**kwargs):
    token = "<your_token>"  # Your Scrape.do API token
    target_url = "https://books.toscrape.com/"

    response = requests.get(
        "https://api.scrape.do/",
        params={"token": token, "url": target_url, "super": "true", "render": "true"},
        timeout=45,
    )
    if response.status_code != 200:
        # Raising makes Airflow retry the task
        raise ValueError(f"Error Code: {response.status_code}")

    print(response.text[:200])
    return "Scraping Completed"


default_args = {
    "owner": "scrape_do",
    "depends_on_past": False,
    "retries": 2,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="scrape_do_universal_pipeline",
    default_args=default_args,
    description="Scheduled scraping template powered by Scrape.do",
    start_date=datetime(2026, 5, 1),
    schedule="@daily",
    catchup=False,
    tags=["scrape.do"],
) as dag:
    run_scraper = PythonOperator(
        task_id="execute_scrape_do_request",
        python_callable=scrape_do_task,
    )

• Replace <your_token> with your API token from your dashboard and target_url with the page you want.
• super=true uses residential and mobile proxies and render=true renders the page in a headless browser. Drop either one for targets that don't need it, since both add credits. See request costs.
• requests encodes the target URL in params for you.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

4. Run the Pipeline

The DAG appears in the dashboard as scrape_do_universal_pipeline without a restart.

• Toggle it ON with the switch on the DAG row.
• Click the Play button to trigger a run now.
• Open the task instance and switch to the Logs tab to watch the request.

From then on it runs once a day. A failed run waits 2 minutes and retries, and the task only fails after the third attempt.

Full tutorial with every step: Web Scraper for Airflow: The Ultimate Scheduled Pipeline Template (Powered by Scrape.do)