# Gumloop - Scrape.do Integration > Source: https://scrape.do/documentation/integrations/gumloop/ Step-by-step guide to add Scrape.do to Gumloop agents as a reusable web scraping skill Scrape.do works in Gumloop as a **skill**: a folder with a `SKILL.md` file and a Python script that any agent in your workspace can load. The agent runs the script in its code sandbox, Scrape.do fetches the page, and the agent reads the result as markdown or HTML. ## 1. Get Your Scrape.do API Token - Sign up at [dashboard.scrape.do](https://dashboard.scrape.do) if you don't have an account. - Copy your API token from the dashboard. The free plan is enough to follow this guide end to end. ## 2. Store the Token as a Secret Never paste the token into a skill file. Skills get shared and exported, and anything written inside them travels with the file. Gumloop's Vault keeps it out. - Go to **Settings** and open your personal **Vault**, or the team Vault if the whole team will use this skill. - Add a new secret and paste your Scrape.do token into it. - Open the agent that will use the skill, go to its **Agent** tab, and expand **Secrets**. - Click **+ Add** and configure: - **Name** - Enter `SCRAPE_DO_TOKEN`. This is the environment variable the script reads. - **Source** - Select the Vault secret you just created. - Click **Add**. The agent can reference the secret by name but never sees the raw value. At run time Gumloop injects it into the code sandbox as an environment variable. ## 3. Create the Skill - Go to the **Skills** page and create a new skill. - Name it `scrape-do-web-scraping`. A skill is a folder built around a single `SKILL.md` file. It can also hold three optional directories: ```plaintext scrape-do-web-scraping/ SKILL.md scripts/ scrape.py references/ assets/ ``` Only `SKILL.md` is required. You'll add one script in step 5. ## 4. Write the SKILL.md `SKILL.md` is Markdown with a YAML header. The header tells Gumloop what the skill is called and when to load it. The body tells the agent how to use it. ```markdown --- name: scrape-do-web-scraping description: >- Fetch any public web page through the Scrape.do API and return its content as clean markdown or as specific fields. Use this whenever a page needs to be read and an ordinary request would be blocked, rate limited, or served a CAPTCHA. icon: globe color: "#F9D94A" --- # Scrape.do Web Scraping Use this skill to read the contents of any public web page. Scrape.do handles proxy rotation, browser rendering, and anti-bot challenges, so a page that blocks a plain request still comes back successfully. ## When to use this - The page returned a 403, a CAPTCHA, or an empty body on a direct request - The page builds its content with JavaScript - You need many pages from the same site without being blocked - The site serves different content by country ## How to run it Run the script in `scripts/scrape.py` from the sandbox shell: python scripts/scrape.py "" Options: --output markdown Clean markdown, best for reading and summarizing (default) --output raw Full HTML, use when you need specific elements --render Turn on browser rendering for JavaScript heavy pages --super Use residential and mobile proxies on protected sites --geo-code us Two letter country code for the proxy location ## Choosing the options Start with no flags. Markdown output through datacenter proxies is the cheapest and fastest path, and it works for most sites. Add `--render` only when the returned content is missing what you can see in a browser. Add `--super` only when the request fails without it. Each flag costs more credits, so adding both by default wastes budget on pages that did not need them. ## Reading the result The script prints the page content to standard output. For markdown output, read it directly. For raw HTML, parse what you need rather than passing the whole document into context, since a single product page can run past 500KB. ``` The `description` field decides whether the agent loads the skill at all, so write it as a trigger condition, not a summary. "Fetch any public web page" says what the skill does. "Whenever a page would be blocked, rate limited, or served a CAPTCHA" says when to reach for it, and that half is what makes the skill fire at the right moment. ## 5. Add the Scraping Script Create `scripts/scrape.py` inside the skill folder. The Gumloop code sandbox ships with over 80 Python packages, including `requests`, and has full outbound internet access, so no setup is needed. ```python import argparse import os import sys import requests API_URL = "https://api.scrape.do/" def scrape(url, output="markdown", render=False, super_proxy=False, geo_code="us"): token = os.environ.get("SCRAPE_DO_TOKEN") if not token: sys.exit("SCRAPE_DO_TOKEN is not set. Add it under the agent's Secrets.") params = { "token": token, "url": url, "output": output, "render": str(render).lower(), "super": str(super_proxy).lower(), "geoCode": geo_code, } response = requests.get(API_URL, params=params, timeout=120) if response.status_code == 401 or response.status_code == 403: sys.exit("Scrape.do rejected the token. Check the secret value.") if response.status_code == 429: sys.exit("Plan concurrency limit reached. Retry in a moment.") if response.status_code in (502, 503): sys.exit("The proxy could not reach the target. Retry, or add --super.") response.raise_for_status() return response.text def main(): parser = argparse.ArgumentParser(description="Scrape a page via Scrape.do") parser.add_argument("url") parser.add_argument("--output", default="markdown", choices=["markdown", "raw"]) parser.add_argument("--render", action="store_true") parser.add_argument("--super", dest="super_proxy", action="store_true") parser.add_argument("--geo-code", default="us") args = parser.parse_args() print(scrape( args.url, output=args.output, render=args.render, super_proxy=args.super_proxy, geo_code=args.geo_code, )) if __name__ == "__main__": main() ``` - The script passes the parameters as a dictionary and lets `requests` build the query string. The target URL carries its own `?` and `&` characters, and `requests` encodes it correctly, so you don't need a manual encoding step. - `--geo-code` takes any supported country code (`us`, `gb`, `de`...). View available locations [here](/documentation/proxies/geo-code). - A `401` also comes back when the account is out of credits. See [status codes](/documentation/api-response/status-codes) for the rest. Save the skill. Gumloop stores the folder and makes it available to any agent you attach it to. ## 6. Give the Skill to an Agent - Open the agent you want to use it. - Find the **Skills** section and add `scrape-do-web-scraping`. - Confirm that the `SCRAPE_DO_TOKEN` secret from step 2 is bound on the same agent. The skill and the secret are attached separately. A skill without its secret saves fine and then fails at run time with a missing token. This is the most common setup mistake. ## 7. Test the Agent Give the agent a task that needs a page it can't fetch on its own: ```plaintext Read https://us.amazon.com/dp/B0BLRJ4R8F and tell me the product name, the current price, and the review rating. ``` Watch the run and check that: - The agent loaded the skill rather than trying a direct request first - The script ran without a token error - The returned markdown contains the product details - The agent answered from the scraped content rather than from guesswork If the agent tries a plain fetch first and gives up, the `description` in `SKILL.md` isn't specific enough about the blocked-page case. Tighten it and test again. > If you're unable to get a response, visit the [playground](https://dashboard.scrape.do/playground) and experiment with parameters like **Super**, **Render JavaScript**, and **Block Resources** until you get a successful response. Then pass the same options to the script: **Super** is `--super` and **Render JavaScript** is `--render`. Other parameters need a new entry in the script's `params`. Each flag costs more credits per request. See [request costs](/documentation/request-costs) for the full table. ## 8. Share and Reuse the Skill - From the three-dot menu on the **Skills** page, download the skill. You get a `.zip` or `.skill` file containing `SKILL.md` and your script. - Upload that file into any other Gumloop workspace to recreate the skill exactly. Uploads accept `.md`, `.zip`, and `.skill`. - Share it directly by email, with your team, with your whole organization, or with anyone who has the link. - Assign roles per person: **Owner**, **Editor**, **Viewer**, or **Use Only**. **Use Only** lets teammates run the skill in their agents without seeing its files. The only thing each person supplies is their own token. The skill reads it from `SCRAPE_DO_TOKEN`, so whoever imports the file adds their own secret under that name and the skill works unchanged.