KNIME - Scrape.do Integration
Step-by-step guide to integrate Scrape.do with KNIME for no-code web scraping workflows
Scrape.do integrates with KNIME through a free workflow template. You add your token, paste your URLs, and the workflow sends each one through Scrape.do's rotating proxies and returns the HTML, with no code to write.
Download the KNIME workflow template (.knwf) to follow along.
1. Import the Workflow
Open KNIME Analytics Platform and go to File → Import KNIME Workflow.... Select the downloaded .knwf file and the scraping workflow appears on your canvas.
2. Add Your API Token
Double-click the first Table Creator node (the one labeled for your Token). Replace ENTER_YOUR_TOKEN_HERE with your Scrape.do API token from your dashboard and click OK. You only do this once.
3. Add Your Target URLs
Double-click the second Table Creator node. In the URL column, paste the pages you want to scrape and click OK.
4. Run the Workflow
Click the green Play button in the top toolbar (or press Shift + F7) to execute all nodes.
The workflow merges your token and URLs and sends them to the GET Request node, which routes each request through Scrape.do. The Row Filter node drops failed requests, so only Status 200 responses pass through.
If you're unable to get a response, visit the playground and experiment with parameters like Super, Render JavaScript, and Block Resources until you get a successful response.
5. Extract the Fields You Need
Double-click the Regex Split node at the end of the chain and write a regular expression in the Pattern field. For example, to pull the page title and the price from an Amazon product page:
(?s).*?<title>(.*?)</title>.*?class="a-price-whole">([^<]+).*The text inside <title> goes into one column and the number inside the a-price-whole class goes into another. Right-click the node and open Split Result to see the output.
Full tutorial with every step: How to Scrape Any Website in KNIME (Free Workflow Template)
