Gumloop - Scrape.do Integration | Documentation

Step-by-step guide to add Scrape.do to Gumloop agents as a reusable web scraping skill

Scrape.do works in Gumloop as a skill: a folder with a SKILL.md file and a Python script that any agent in your workspace can load. The agent runs the script in its code sandbox, Scrape.do fetches the page, and the agent reads the result as markdown or HTML.

1. Get Your Scrape.do API Token

Sign up at dashboard.scrape.do if you don't have an account.
Copy your API token from the dashboard. The free plan is enough to follow this guide end to end.

2. Store the Token as a Secret

Never paste the token into a skill file. Skills get shared and exported, and anything written inside them travels with the file. Gumloop's Vault keeps it out.

Go to Settings and open your personal Vault, or the team Vault if the whole team will use this skill.
Add a new secret and paste your Scrape.do token into it.
Open the agent that will use the skill, go to its Agent tab, and expand Secrets.
Click + Add and configure:
- Name - Enter SCRAPE_DO_TOKEN. This is the environment variable the script reads.
- Source - Select the Vault secret you just created.
Click Add.

The agent can reference the secret by name but never sees the raw value. At run time Gumloop injects it into the code sandbox as an environment variable.

3. Create the Skill

Go to the Skills page and create a new skill.
Name it scrape-do-web-scraping.

A skill is a folder built around a single SKILL.md file. It can also hold three optional directories:

scrape-do-web-scraping/
  SKILL.md
  scripts/
    scrape.py
  references/
  assets/

Only SKILL.md is required. You'll add one script in step 5.

4. Write the SKILL.md

SKILL.md is Markdown with a YAML header. The header tells Gumloop what the skill is called and when to load it. The body tells the agent how to use it.

---
name: scrape-do-web-scraping
description: >-
  Fetch any public web page through the Scrape.do API and return its content as
  clean markdown or as specific fields. Use this whenever a page needs to be
  read and an ordinary request would be blocked, rate limited, or served a
  CAPTCHA.
icon: globe
color: "#F9D94A"
---

# Scrape.do Web Scraping

Use this skill to read the contents of any public web page. Scrape.do handles
proxy rotation, browser rendering, and anti-bot challenges, so a page that
blocks a plain request still comes back successfully.

## When to use this

- The page returned a 403, a CAPTCHA, or an empty body on a direct request
- The page builds its content with JavaScript
- You need many pages from the same site without being blocked
- The site serves different content by country

## How to run it

Run the script in `scripts/scrape.py` from the sandbox shell:

    python scripts/scrape.py "<url>"

Options:

    --output markdown   Clean markdown, best for reading and summarizing (default)
    --output raw        Full HTML, use when you need specific elements
    --render            Turn on browser rendering for JavaScript heavy pages
    --super             Use residential and mobile proxies on protected sites
    --geo-code us       Two letter country code for the proxy location

## Choosing the options

Start with no flags. Markdown output through datacenter proxies is the
cheapest and fastest path, and it works for most sites.

Add `--render` only when the returned content is missing what you can see in a
browser. Add `--super` only when the request fails without it. Each flag costs
more credits, so adding both by default wastes budget on pages that did not
need them.

## Reading the result

The script prints the page content to standard output. For markdown output,
read it directly. For raw HTML, parse what you need rather than passing the
whole document into context, since a single product page can run past 500KB.

The description field decides whether the agent loads the skill at all, so write it as a trigger condition, not a summary. "Fetch any public web page" says what the skill does. "Whenever a page would be blocked, rate limited, or served a CAPTCHA" says when to reach for it, and that half is what makes the skill fire at the right moment.

5. Add the Scraping Script

Create scripts/scrape.py inside the skill folder. The Gumloop code sandbox ships with over 80 Python packages, including requests, and has full outbound internet access, so no setup is needed.

import argparse
import os
import sys

import requests

API_URL = "https://api.scrape.do/"


def scrape(url, output="markdown", render=False, super_proxy=False, geo_code="us"):
    token = os.environ.get("SCRAPE_DO_TOKEN")
    if not token:
        sys.exit("SCRAPE_DO_TOKEN is not set. Add it under the agent's Secrets.")

    params = {
        "token": token,
        "url": url,
        "output": output,
        "render": str(render).lower(),
        "super": str(super_proxy).lower(),
        "geoCode": geo_code,
    }

    response = requests.get(API_URL, params=params, timeout=120)

    if response.status_code == 401 or response.status_code == 403:
        sys.exit("Scrape.do rejected the token. Check the secret value.")
    if response.status_code == 429:
        sys.exit("Plan concurrency limit reached. Retry in a moment.")
    if response.status_code in (502, 503):
        sys.exit("The proxy could not reach the target. Retry, or add --super.")

    response.raise_for_status()
    return response.text


def main():
    parser = argparse.ArgumentParser(description="Scrape a page via Scrape.do")
    parser.add_argument("url")
    parser.add_argument("--output", default="markdown", choices=["markdown", "raw"])
    parser.add_argument("--render", action="store_true")
    parser.add_argument("--super", dest="super_proxy", action="store_true")
    parser.add_argument("--geo-code", default="us")
    args = parser.parse_args()

    print(scrape(
        args.url,
        output=args.output,
        render=args.render,
        super_proxy=args.super_proxy,
        geo_code=args.geo_code,
    ))


if __name__ == "__main__":
    main()

The script passes the parameters as a dictionary and lets requests build the query string. The target URL carries its own ? and & characters, and requests encodes it correctly, so you don't need a manual encoding step.
--geo-code takes any supported country code (us, gb, de...). View available locations here.
A 401 also comes back when the account is out of credits. See status codes for the rest.

Save the skill. Gumloop stores the folder and makes it available to any agent you attach it to.

6. Give the Skill to an Agent

Open the agent you want to use it.
Find the Skills section and add scrape-do-web-scraping.
Confirm that the SCRAPE_DO_TOKEN secret from step 2 is bound on the same agent.

The skill and the secret are attached separately. A skill without its secret saves fine and then fails at run time with a missing token. This is the most common setup mistake.

7. Test the Agent

Give the agent a task that needs a page it can't fetch on its own:

Read https://us.amazon.com/dp/B0BLRJ4R8F and tell me the product name,
the current price, and the review rating.

Watch the run and check that:

The agent loaded the skill rather than trying a direct request first
The script ran without a token error
The returned markdown contains the product details
The agent answered from the scraped content rather than from guesswork

If the agent tries a plain fetch first and gives up, the description in SKILL.md isn't specific enough about the blocked-page case. Tighten it and test again.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response. Then pass the same options to the script: Super is --super and Render JavaScript is --render. Other parameters need a new entry in the script's params.

Each flag costs more credits per request. See request costs for the full table.

From the three-dot menu on the Skills page, download the skill. You get a .zip or .skill file containing SKILL.md and your script.
Upload that file into any other Gumloop workspace to recreate the skill exactly. Uploads accept .md, .zip, and .skill.
Share it directly by email, with your team, with your whole organization, or with anyone who has the link.
Assign roles per person: Owner, Editor, Viewer, or Use Only. Use Only lets teammates run the skill in their agents without seeing its files.

The only thing each person supplies is their own token. The skill reads it from SCRAPE_DO_TOKEN, so whoever imports the file adds their own secret under that name and the skill works unchanged.