Categories:Scraping Use Cases,Scraping Tools

Web Scraping in Databricks: Land Any Website in a Delta Table with Scrape.do

Clock8 Mins Read
calendarCreated Date: July 14, 2026
calendarUpdated Date: July 28, 2026

Your pipelines live in Databricks. A lot of the data you want doesn't. It's sitting on websites, product listings, prices, search results, public directories, in a form no upstream table will ever hand you. So the obvious move is to import requests in a notebook and start pulling pages down directly into the lakehouse.

That works for about five minutes. Then the blocks start. Your cluster speaks to the world through one IP address, and a website notices very quickly when a single IP asks for its 500th page in a row. What follows is the usual wall: throttling, CAPTCHAs, and JavaScript-rendered pages that return an HTML skeleton with none of the data you came for.

This guide takes the requests off your cluster's shoulders. You'll route them through Scrape.do instead, and land the results directly in a Delta table, ready for the SQL, ML, and AI workloads already sitting downstream. Everything here runs on Databricks Free Edition (serverless), so you can follow along without a paid workspace, and there's a genuinely interesting architecture decision waiting in the middle of it that most "scraping in Spark" tutorials get wrong.

The shape of the notebook

Four steps, each one cell:

  1. Take your Scrape.do token through a notebook widget, never hardcoded.
  2. Wrap the Scrape.do API in a reusable scrape_do() helper with automatic retries.
  3. Fetch a list of URLs in parallel with ThreadPoolExecutor.
  4. Write the results, URL, status, raw HTML, error, timestamp, into a Delta table.

By the end you have a table you can query, parse, join, or feed to a model, refreshed by re-running the notebook or by attaching a schedule to it.

Why the requests go through Scrape.do

Scrape.do is a web scraping API: send it a target URL, get the page content back. The parts that break a cluster-side scraper, it absorbs on its side.

  • Rotating proxies mean each request can leave from a different IP, including residential and mobile ones for stubborn targets with super=true, instead of your cluster's single fixed address.
  • Anti-bot bypass and CAPTCHA handling happen server-side, invisible to your notebook code.
  • JavaScript rendering kicks in with render=true, a real headless browser draws the page before the HTML comes back.

Two economics worth pinning down before you run anything: you're only charged for successful (2xx) responses, so the dead links and transient blocks you hit while testing cost nothing, and the free tier is 1,000 requests a month, more than enough for this whole tutorial. Sign up here.

Step 1 — Spin up the workspace

  1. Head to databricks.com/learn/free-edition and sign up (Google/Microsoft login or email OTP).
  2. When the workspace opens, click + New → Notebook in the top-left.
  3. In the notebook's top-right, confirm the compute selector reads Serverless. Free Edition is serverless-only, and that's all this needs.

Step 2 — Token widget and a connectivity check

Paste this into the first cell and run it:

import requests
import urllib.parse

# Token widget — safe to share the notebook, every user supplies their own token
dbutils.widgets.text("scrapedo_token", "", "Scrape.do API Token")
TOKEN = dbutils.widgets.get("scrapedo_token")

# Quick connectivity test
target = "https://httpbin.org/ip"
url = f"https://api.scrape.do/?token={TOKEN}&url={urllib.parse.quote_plus(target)}"
r = requests.get(url, timeout=60)
print("Status:", r.status_code)
print(r.text[:500])

After the first run, a text box labeled Scrape.do API Token appears at the top of the notebook. Paste your token into that box, not into the code, then run the cell again. Widget values are read on the next execution, which is a quirk worth remembering before you wonder why TOKEN looked empty the first time.

A Status: 200 with a JSON body carrying an IP address proves the two things this entire integration rests on: your serverless compute has outbound internet access, and your token works. Grab that token from the Scrape.do dashboard, it's on the main screen right after login.

Step 3 — The scrape_do() helper

Second cell:

import time

def scrape_do(target_url, render=False, super_proxy=False, max_retries=3):
    """Scrape a URL via Scrape.do. Returns dict with url, status, html, error."""
    params = {"token": TOKEN, "url": target_url}
    if render:
        params["render"] = "true"
    if super_proxy:
        params["super"] = "true"

    api_url = "https://api.scrape.do/?" + urllib.parse.urlencode(params)

    for attempt in range(1, max_retries + 1):
        try:
            r = requests.get(api_url, timeout=120)
            if r.status_code == 200:
                return {"url": target_url, "status": 200, "html": r.text, "error": None}
            if r.status_code == 502 and attempt < max_retries:
                time.sleep(2 * attempt)  # transient ROTATION_FAILED — not charged, retry
                continue
            return {"url": target_url, "status": r.status_code, "html": None, "error": r.text[:300]}
        except requests.exceptions.RequestException as e:
            if attempt < max_retries:
                time.sleep(2 * attempt)
                continue
            return {"url": target_url, "status": None, "html": None, "error": str(e)[:300]}

# Test
result = scrape_do("https://books.toscrape.com/")
print("Status:", result["status"])
print("HTML length:", len(result["html"]) if result["html"] else 0)

Expected output:

Status: 200
HTML length: 51294

Three details in that helper earn their place. urllib.parse.urlencode handles the encoding for you, skip it and Scrape.do rejects the request with "Your target 'URL' is not valid." A transient 502 ROTATION_FAILED happens when a proxy hop fails; it's not charged, so the helper just waits and retries rather than giving up, and if a target keeps failing, super_proxy=True routes it through residential and mobile IPs. And render and super_proxy default off because both cost extra credits, you turn them on per call, only for the targets that need them.

Step 4 — Parallel fetch straight into Delta

Third cell:

from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from datetime import datetime, timezone

# URLs to scrape — replace with your own targets
urls = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
    "https://books.toscrape.com/catalogue/page-4.html",
    "https://books.toscrape.com/catalogue/page-5.html",
]

# Parallel scrape (5 concurrent requests)
with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(scrape_do, urls))

scraped_at = datetime.now(timezone.utc).isoformat()

rows = [
    Row(
        url=r["url"],
        status=r["status"],
        html=r["html"],
        error=r["error"],
        scraped_at=scraped_at,
    )
    for r in results
]

# Define the schema explicitly — don't let Spark infer it
schema = StructType([
    StructField("url", StringType(), False),
    StructField("status", IntegerType(), True),
    StructField("html", StringType(), True),
    StructField("error", StringType(), True),
    StructField("scraped_at", StringType(), False),
])

df = spark.createDataFrame(rows, schema=schema)

# Write to a Delta table
df.write.mode("append").saveAsTable("workspace.default.scraped_pages")

print(f"Scraped {len(rows)} URLs — {sum(1 for r in results if r['status'] == 200)} success")
display(spark.table("workspace.default.scraped_pages").select("url", "status", "scraped_at"))

Expected output:

Scraped 5 URLs — 5 success

…then the table rows: one per page, with its URL, HTTP status, and timestamp. The raw HTML rides in the html column of the same table, so from here it's an ordinary Delta table, parse it with SQL functions, hand it to an LLM, or join it against the rest of your lakehouse. Each run appends, so scheduled re-scrapes accumulate a history automatically; switch to mode("overwrite") if you only ever want the latest snapshot.

The architecture decision worth understanding

Here's where most "scraping in Spark" tutorials steer you wrong. The instinctive Spark move is to wrap scrape_do in a UDF and fan the scraping out across executors, that's the whole point of Spark, right, distribute the work.

On serverless compute, which is all Free Edition offers, that approach doesn't merely underperform. It cannot run: serverless UDFs have no outbound internet access. The executors are sandboxed off the network. The driver, on the other hand, does have internet. So the driver-side ThreadPoolExecutor you saw above isn't a workaround for a missing feature, it's the correct architecture for this tier.

And it holds up well beyond Free Edition, because scraping is I/O-bound. The threads spend nearly all their time waiting on the network, not burning CPU, so 5 to 20 workers on a single driver saturate most scraping jobs. On classic compute clusters a UDF-based fan-out can work, but for typical volumes the ThreadPool version is simpler and just as fast, since the bottleneck was never CPU. It was always the network.

The gotchas we actually hit building this

These aren't hypothetical, every one of them cost us a run while writing this notebook.

NameError: name 'TOKEN' is not defined. Serverless sessions reset after idle time and wipe every variable with them. Re-run from the top (or hit Run all). Cell order is load-bearing: widget → helper → pipeline.

[CANNOT_DETERMINE_TYPE] when building the DataFrame. If every scrape succeeds, error is None in every row and Spark can't infer that column's type. That's exactly why the code declares the schema explicitly with StructType. Never let Spark infer a schema for columns that can come back all-null.

[SCHEMA_NOT_FOUND] The schema main.default cannot be found. Databricks docs love main as the catalog name, but your workspace's default catalog may differ, on our Free Edition workspace it was workspace. Check yours:

display(spark.sql("SHOW CATALOGS"))
print(spark.sql("SELECT current_catalog(), current_schema()").collect())

Then use <your_catalog>.default.scraped_pages as the table name.

Transient 502 ROTATION_FAILED. A proxy hop occasionally fails on Scrape.do's side. It isn't charged and a retry almost always clears it, the helper handles this. Persistent 502s on one domain mean the target is genuinely hard; reach for super_proxy=True.

Widget values read on the next run. Run the first cell, paste the token into the widget box, run it again. If TOKEN is empty, this is why.

Store raw HTML deliberately. A Delta string column happily holds full-page HTML (50 KB+ per row), unlike spreadsheet cells that truncate. But if you only need a few fields, parse them before writing and keep the table lean, and keep input columns (URLs) strictly separate from output columns (HTML) so a re-run never feeds scraped output back in as input.

Run order and scheduling

  1. Cell 1 — widget + connectivity test (paste the token, run twice on first use).
  2. Cell 2 — the scrape_do() helper.
  3. Cell 3 — parallel scrape and Delta write.

Once it's set up, Run all executes the whole pipeline in a click, and the Schedule button in the notebook toolbar turns it into a recurring scraping job with no extra code. That's the payoff of landing scraped data in the lakehouse directly: the scheduling, retries, and downstream analytics you already have all apply to web data the moment it arrives.

Start scraping

Everything above runs on free tiers end to end, Databricks Free Edition for compute, and 1,000 free requests a month from Scrape.do for the scraping, with proxy rotation, anti-bot bypass, CAPTCHA solving, and JavaScript rendering included, billed only for the responses that actually succeed.

👉 Get your free Scrape.do API token and turn any website into a Delta table.

Downloads:

Table of Contents

Reliable, Scalable, Unstoppable Web Scraping

START FOR FREE