Alteryx - Scrape.do Integration | Documentation

Step-by-step guide to integrate Scrape.do with Alteryx Designer using a ready-made scraping macro

Scrape.do works in Alteryx Designer as a macro. Feed it a column of URLs, get each page's HTML back in a DownloadData column, then parse the fields you need with the RegEx tool.

1. Insert the Macro

Download the Scrape.do macro (.yxmc).

Open a blank workflow in Alteryx Designer.
Right-click the canvas, select Insert and click Macro....
Select the .yxmc file you downloaded. The Scrape.do macro drops onto the canvas.

2. Add Your Scrape.do Token

Right-click the macro and select Open Macro.
Click the green Text Input tool at the start of the macro.
Replace INSERT_YOUR_SCRAPEDO_API_TOKEN with your API token from your dashboard.
Save with Ctrl + S and close the macro tab. You only do this once.

3. Feed the URLs

Drag a Text Input tool onto your main workflow, to the left of the macro.
Create a column named url and paste the pages you want to scrape.
Connect the Text Input output to the macro's input anchor.

4. Run the Scraper

Add a Browse tool to the macro's output and click Run. The DownloadData column holds the HTML of each page.

The macro sends every request with super=true and render=true, so each page goes through residential proxies and a headless browser. See request costs for what that uses.

If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.

5. Extract Data with RegEx

Drag a RegEx tool onto the canvas and connect it to the macro's output.
Set Field to Parse to DownloadData and Output Method to Parse.
Enter a pattern. For example, <title>(.*?)</title> puts the page title in a new column, and class="price">([^<]+) pulls a price from <span class="price">99.99</span>.