# Databricks - Scrape.do Integration

> Source: https://scrape.do/documentation/integrations/databricks/

Step-by-step guide to integrate Scrape.do with Databricks and land scraped pages in a Delta table

Scrape.do integrates with Databricks from a notebook. Your requests go through Scrape.do's rotating proxies instead of your cluster's single IP, and the results land in a Delta table. Everything below runs on Databricks Free Edition (serverless).

**<a href="/uploads/blog/scrapedo-databricks-notebook.ipynb" download="scrapedo-databricks-notebook.ipynb">Download the Databricks notebook</a>** to import it into your workspace.

## 1. Create a Notebook

- Sign up at [Databricks Free Edition](https://www.databricks.com/learn/free-edition), or use your existing workspace.
- Click **+ New → Notebook**.
- In the notebook's top-right, confirm the compute selector reads **Serverless**.

## 2. Add Your Token and Test the Connection

Paste this into the first cell and run it:

```python
import requests
import urllib.parse

# Token widget: every user supplies their own token
dbutils.widgets.text("scrapedo_token", "", "Scrape.do API Token")
TOKEN = dbutils.widgets.get("scrapedo_token")

# Quick connectivity test
target = "https://httpbin.org/ip"
url = f"https://api.scrape.do/?token={TOKEN}&url={urllib.parse.quote_plus(target)}"
r = requests.get(url, timeout=60)
print("Status:", r.status_code)
print(r.text[:500])
```

A **Scrape.do API Token** text box appears at the top of the notebook. Paste your token from your [dashboard](https://dashboard.scrape.do) into that box, not into the code, and run the cell again. Widget values are read on the next run. `Status: 200` means your compute can reach the internet and your token works.

## 3. Add the Scraping Helper

Second cell:

```python
import time

def scrape_do(target_url, render=False, super_proxy=False, max_retries=3):
    """Scrape a URL via Scrape.do. Returns dict with url, status, html, error."""
    params = {"token": TOKEN, "url": target_url}
    if render:
        params["render"] = "true"
    if super_proxy:
        params["super"] = "true"

    api_url = "https://api.scrape.do/?" + urllib.parse.urlencode(params)

    for attempt in range(1, max_retries + 1):
        try:
            r = requests.get(api_url, timeout=120)
            if r.status_code == 200:
                return {"url": target_url, "status": 200, "html": r.text, "error": None}
            if r.status_code == 502 and attempt < max_retries:
                time.sleep(2 * attempt)  # transient ROTATION_FAILED, not charged, retry
                continue
            return {"url": target_url, "status": r.status_code, "html": None, "error": r.text[:300]}
        except requests.exceptions.RequestException as e:
            if attempt < max_retries:
                time.sleep(2 * attempt)
                continue
            return {"url": target_url, "status": None, "html": None, "error": str(e)[:300]}
```

`urllib.parse.urlencode` encodes the target URL for you. Without it, Scrape.do rejects the request with `Your target 'URL' is not valid.` Pass `render=True` for JavaScript-heavy pages and `super_proxy=True` for targets that block datacenter IPs. Both cost extra credits, so turn them on only where needed.

> If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. Then pass the same options to `scrape_do()`: **Super** is `super_proxy=True` and **Render JavaScript** is `render=True`.

## 4. Scrape in Parallel and Write to Delta

Third cell:

```python
from concurrent.futures import ThreadPoolExecutor
from pyspark.sql import Row
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from datetime import datetime, timezone

# URLs to scrape: replace with your own targets
urls = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
]

with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(scrape_do, urls))

scraped_at = datetime.now(timezone.utc).isoformat()
rows = [
    Row(url=r["url"], status=r["status"], html=r["html"], error=r["error"], scraped_at=scraped_at)
    for r in results
]

# Define the schema explicitly so all-null columns don't break inference
schema = StructType([
    StructField("url", StringType(), False),
    StructField("status", IntegerType(), True),
    StructField("html", StringType(), True),
    StructField("error", StringType(), True),
    StructField("scraped_at", StringType(), False),
])

df = spark.createDataFrame(rows, schema=schema)
df.write.mode("append").saveAsTable("workspace.default.scraped_pages")

display(spark.table("workspace.default.scraped_pages").select("url", "status", "scraped_at"))
```

Each run appends to the table, so scheduled runs build a history. Use `mode("overwrite")` if you only want the latest snapshot. If your default catalog isn't `workspace`, check it with `SHOW CATALOGS` and use `<your_catalog>.default.scraped_pages`.

Keep the fetching on the driver as shown. On serverless compute, UDFs have no outbound internet access, so a UDF-based scraper can't reach Scrape.do.

## 5. Schedule It

Click **Run all** to run the three cells in order. To re-scrape on a schedule, use the **Schedule** button in the notebook toolbar.

Full tutorial with every step: [Web Scraping in Databricks: Land Any Website in a Delta Table with Scrape.do](/blog/databricks-web-scraping/)