Databricks - Scrape.do Integration | Documentation

Step-by-step guide to integrate Scrape.do with Databricks and land scraped pages in a Delta table

Scrape.do integrates with Databricks from a notebook. Your requests go through Scrape.do's rotating proxies instead of your cluster's single IP, and the results land in a Delta table. Everything below runs on Databricks Free Edition (serverless).

Download the Databricks notebook to import it into your workspace.

1. Create a Notebook

Sign up at Databricks Free Edition, or use your existing workspace.
Click + New → Notebook.
In the notebook's top-right, confirm the compute selector reads Serverless.

2. Add Your Token and Test the Connection

Paste this into the first cell and run it:

import requests
import urllib.parse

# Token widget: every user supplies their own token
dbutils.widgets.text("scrapedo_token", "", "Scrape.do API Token")
TOKEN = dbutils.widgets.get("scrapedo_token")

# Quick connectivity test
target = "https://httpbin.org/ip"
url = f"https://api.scrape.do/?token={TOKEN}&url={urllib.parse.quote_plus(target)}"
r = requests.get(url, timeout=60)
print("Status:", r.status_code)
print(r.text[:500])

A Scrape.do API Token text box appears at the top of the notebook. Paste your token from your dashboard into that box, not into the code, and run the cell again. Widget values are read on the next run. Status: 200 means your compute can reach the internet and your token works.

3. Add the Scraping Helper

Second cell:

import time

def scrape_do(target_url, render=False, super_proxy=False, max_retries=3):
    """Scrape a URL via Scrape.do. Returns dict with url, status, html, error."""
    params = {"token": TOKEN, "url": target_url}
    if render:
        params["render"] = "true"
    if super_proxy:
        params["super"] = "true"

    api_url = "https://api.scrape.do/?" + urllib.parse.urlencode(params)

    for attempt in range(1, max_retries + 1):
        try:
            r = requests.get(api_url, timeout=120)
            if r.status_code == 200:
                return {"url": target_url, "status": 200, "html": r.text, "error": None}
            if r.status_code == 502 and attempt < max_retries:
                time.sleep(2 * attempt)  # transient ROTATION_FAILED, not charged, retry
                continue
            return {"url": target_url, "status": r.status_code, "html": None, "error": r.text[:300]}
        except requests.exceptions.RequestException as e:
            if attempt < max_retries:
                time.sleep(2 * attempt)
                continue
            return {"url": target_url, "status": None, "html": None, "error": str(e)[:300]}

urllib.parse.urlencode encodes the target URL for you. Without it, Scrape.do rejects the request with Your target 'URL' is not valid. Pass render=True for JavaScript-heavy pages and super_proxy=True for targets that block datacenter IPs. Both cost extra credits, so turn them on only where needed.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Then pass the same options to scrape_do(): Super is super_proxy=True and Render JavaScript is render=True.

4. Scrape in Parallel and Write to Delta

Third cell:

from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from datetime import datetime, timezone

# URLs to scrape: replace with your own targets
urls = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
]

with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(scrape_do, urls))

scraped_at = datetime.now(timezone.utc).isoformat()
rows = [
    Row(url=r["url"], status=r["status"], html=r["html"], error=r["error"], scraped_at=scraped_at)
    for r in results
]

# Define the schema explicitly so all-null columns don't break inference
schema = StructType([
    StructField("url", StringType(), False),
    StructField("status", IntegerType(), True),
    StructField("html", StringType(), True),
    StructField("error", StringType(), True),
    StructField("scraped_at", StringType(), False),
])

df = spark.createDataFrame(rows, schema=schema)
df.write.mode("append").saveAsTable("workspace.default.scraped_pages")

display(spark.table("workspace.default.scraped_pages").select("url", "status", "scraped_at"))

Each run appends to the table, so scheduled runs build a history. Use mode("overwrite") if you only want the latest snapshot. If your default catalog isn't workspace, check it with SHOW CATALOGS and use <your_catalog>.default.scraped_pages.

Keep the fetching on the driver as shown. On serverless compute, UDFs have no outbound internet access, so a UDF-based scraper can't reach Scrape.do.