Advanced Web Scraping Techniques with Puppeteer and NodeJS
Web scraping allows you to gather data from websites efficiently, whether for extracting product information, monitoring page changes, or visualizing time-based data. In this guide, we leverage Puppeteer, a powerful headless browser automation tool, together with Node.js to manage asynchronous tasks and enhance web scraping capabilities. We’ll also integrate Scrape.do to optimize our scraping process with advanced features like proxy management, session handling, and bot detection avoidance.
NodeJS and Puppeteer
Puppeteer is an open-source Node.js library that provides a high-level API for controlling Chrome or Chromium over the DevTools Protocol. It enables developers to automate browsing tasks, such as clicking buttons, filling out forms, and capturing screenshots, while navigating websites as a real user would. By simulating real-user interactions, Puppeteer is particularly effective at scraping dynamic content generated by JavaScript, which traditional scraping tools may fail to capture.
Node.js, also called NodeJS, is a popular runtime environment that features an event-driven architecture essential for managing multiple web scraping tasks simultaneously. Together, Puppeteer and Node.js provide a robust platform for building scalable and efficient web scrapers capable of handling various challenges, including dynamic content loading, CAPTCHA challenges, and bot detection.
Environment Setup
Dependencies
To get started, you’ll need to install Node.js and Puppeteer. Make sure you are using stable versions for the web scraping tasks that follow.
npm install puppeteer@latest
Configuration
Set up a basic Puppeteer project with Node.js. First, create a package.json file by running the following command:
npm init -y
Then, configure your package.json to include the necessary dependencies:
{
"name": "web-scraper",
"version": "1.0.0",
"description": "Advanced web scraping with Puppeteer, NodeJS, and Scrape.do",
"main": "index.js",
"dependencies": {
"puppeteer": "^19.9.2",
"axios": "^1.5.0",
}
}
Advanced Cookie and Session Management
Efficient management of cookies and sessions is essential in web scraping to maintain continuity and avoid repetitive logins or verification checks. When handling form data that appears in a modal triggered by a button, it is crucial to ensure the session state is preserved throughout the scraping process. To achieve this, we will create cookies with timestamps, check for existing cookies, and reuse them as needed. This helps maintain the login state across multiple requests and reduces the likelihood of being detected and blocked by the target website.
Cookies store important information, such as login credentials, user preferences, and session IDs, which are vital for maintaining the login state and session data during scraping. Without proper session management, a scraper may trigger repeated login attempts or CAPTCHAs, increasing the risk of detection and blocking.
Session Management
To handle cookies and sessions effectively, we will demonstrate how to save cookies to a file and reload them in subsequent scraping sessions. This approach maintains the login state and minimizes the risk of being flagged as a bot. Additionally, we will use Scrape.do to manage proxies, solve blocking issues, and implement retry mechanisms that enable the scraper to handle CAPTCHAs, detect and bypass rate limits, and manage retries efficiently.
Here is an example of how to manage sessions and cookies to maintain login states across multiple scraping runs:
const puppeteer = require('puppeteer');
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const os = require('os');
const SCRAPE_DO_TOKEN = '--your-token--'; // Replace with your Scrape.do API token
const targetUrl = encodeURIComponent('https://example.com/');
const blockedResourceTypes = [
'beacon',
'csp_report',
'font',
'image',
'imageset',
'media',
'object',
'texttrack',
'stylesheet',
];
// Custom delay function
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
// JSON-like configuration for form fields
const formConfig = [
{ selector: 'input[name="name"]', value: 'John Doe', type: 'text' },
{ selector: 'input[name="email"]', value: '[email protected]', type: 'email' },
{ selector: 'textarea[name="message"]', value: 'Hello, this is a test!', type: 'textarea' },
];
(async () => {
try {
// Step 1: Use Scrape.do API to fetch page content
const response = await axios.get(`https://api.scrape.do?token=${SCRAPE_DO_TOKEN}&url=${targetUrl}`, {
timeout: 30000 // Set timeout for the request
});
console.log('Scrape.do API response received.');
// Step 2: Launch Puppeteer to process the fetched content
const browser = await puppeteer.launch({
headless: false,
ignoreHTTPSErrors: true, // Ignore SSL errors for any certificates
});
const page = await browser.newPage();
// Block unnecessary resources
await page.setRequestInterception(true);
page.on('request', request => {
if (blockedResourceTypes.includes(request.resourceType())) {
console.log(`Blocked type: ${request.resourceType()} url: ${request.url()}`);
request.abort();
} else {
request.continue();
}
});
// Load cookies from a file if available
const cookiePath = 'cookies.json';
let cookies = [];
if (fs.existsSync(cookiePath)) {
cookies = JSON.parse(fs.readFileSync(cookiePath, 'utf-8'));
console.log('Loaded cookies:', cookies);
await page.setCookie(...cookies);
}
// Step 3: Set the content obtained from Scrape.do in Puppeteer
await page.setContent(response.data);
console.log('Page content loaded into Puppeteer.');
// Step 4: Click the button to open the modal
await page.waitForSelector('#open-modal'); // Wait for the button to appear
await page.click('#open-modal'); // Click the button to open the modal
// Wait for the modal to become visible
await page.waitForSelector('.modal-content', { visible: true });
console.log('Modal appeared, checking if the modal has a form...');
// Check if the modal contains a form
const isFormPresent = await page.evaluate(() => {
const modal = document.querySelector('.modal-content');
return modal && modal.querySelector('form#contact-form') !== null; // Check if the form exists in the modal
});
if (!isFormPresent) {
console.error('No form found in the modal.');
await browser.close();
return; // Exit the script if the form is not present
}
console.log('Form is present in the modal.');
// Fill in the form fields dynamically using the configuration
for (const field of formConfig) {
console.log(`Filling field: ${field.selector} with value: ${field.value}`);
await delay(1000); // Wait for 1 second before typing
await page.type(field.selector, field.value);
}
// Wait for the file chooser, then trigger the file input click and upload a file
const [fileChooser] = await Promise.all([
page.waitForFileChooser(),
page.click('input[type="file"]'), // This will trigger the file input
]);
// Select the file to upload from ~/Desktop/test.jpg
const filePath = path.join(os.homedir(), 'Desktop', 'test.jpg'); // Resolve the path to ~/Desktop/test.jpg
await fileChooser.accept([filePath]);
// Click the submit button
await delay(1000);
await page.click('button#submit-form');
// Wait for the confirmation message to appear
await page.waitForSelector('.form-submission-confirmation', { visible: true });
// Extract the form data
const formName = await page.$eval('form#contact-form', form => form.getAttribute('name'));
const submissionUrl = page.url();
// Extract the values of the fields dynamically from the configuration
const extractedFormData = {};
for (const field of formConfig) {
extractedFormData[field.selector] = await page.$eval(field.selector, el => el.value);
}
// Log the data to the console
console.log({
formName,
submissionUrl,
extractedFormData,
});
// Step 5: Save cookies after the form submission
cookies = await page.cookies();
fs.writeFileSync(cookiePath, JSON.stringify(cookies));
console.log('Cookies saved to file.');
await browser.close();
} catch (error) {
console.error('Error during scraping:', error);
// Retry logic using Scrape.do API retry feature
try {
const retryResponse = await axios.post('https://api.scrape.do/retry', { token: SCRAPE_DO_TOKEN });
console.log('Retry triggered with Scrape.do', retryResponse.data);
} catch (retryError) {
console.error('Retry error:', retryError);
}
}
})();
Expected Output
When the script runs successfully, you will see the following output in your terminal:
Scrape.do API response received.
Page content loaded into Puppeteer.
Modal appeared, checking if the modal has a form...
Form is present in the modal.
Filling field: input[name="name"] with value: John Doe
Filling field: input[name="email"] with value: john.doe@example.com
Filling field: textarea[name="message"] with value: Hello, this is a test!
{
formName: null,
submissionUrl: 'about:blank',
extractedFormData: {
'input[name="name"]': 'John Doe',
'input[name="email"]': '[email protected]',
'textarea[name="message"]': 'Hello, this is a test!'
}
}
Cookies saved to file.
Potential Challenges During the Process
When using Puppeteer and Node.js for complex web scraping tasks, some challenges may arise. However, with Scrape.do in API Mode or Proxy Mode, and by leveraging its additional features, it is possible to manage the entire process efficiently.
Increase the Timeout Value
Web scraping often requires handling dynamic content or slow network responses. Increase the timeout value in your Puppeteer and Scrape.do configuration to give more time for requests to complete.
const scrapeDoUrl = `https://api.scrape.do?token=${SCRAPE_DO_TOKEN}&url=https://66d766b074bd9da7360c1769--incandescent-gnome-f2685a.netlify.app&keep_headers=true&timeout=30000`; // 30 seconds timeout
(async () => {
const browser = await puppeteer.launch({ headless: false, ignoreHTTPSErrors: true });
const page = await browser.newPage();
try {
// Increase Puppeteer's timeout
await page.goto(scrapeDoUrl, { waitUntil: 'networkidle2', timeout: 60000 }); // 60 seconds timeout
console.log('Page loaded successfully.');
} catch (error) {
console.error('Error during scraping:', error);
// Retry logic using Scrape.do API retry feature
try {
const retryResponse = await axios.post('https://api.scrape.do/retry', { token: SCRAPE_DO_TOKEN });
console.log('Retry response:', retryResponse.data);
} catch (retryError) {
console.error('Retry error:', retryError);
}
} finally {
await browser.close();
}
})();
Add Exponential Backoff for Retry Logic
Incorporate an exponential backoff strategy to handle retries more efficiently, allowing for progressively increased delays between attempts.
async function retryWithBackoff(fn, maxRetries = 5, delay = 1000) {
for (let i = 0; i < maxRetries; i++) {
try {
return await fn();
} catch (error) {
console.error(`Attempt ${i + 1} failed:`, error.message);
if (i === maxRetries - 1) throw error;
await new Promise(res => setTimeout(res, delay * (i + 1))); // Exponential backoff
}
}
}
retryWithBackoff(() => page.goto(scrapeDoUrl, { waitUntil: 'networkidle2', timeout: 60000 }));
Use Scrape.do’s solve_blocking Parameter
Enable the solve_blocking feature in Scrape.do to solve potential blocking issues that could be causing timeouts.
const scrapeDoUrl = `https://api.scrape.do?token=${SCRAPE_DO_TOKEN}&url=https://66d766b074bd9da7360c1769--incandescent-gnome-f2685a.netlify.app&keep_headers=true&timeout=30000&solve_blocking=true`;
Optimizing Performance Through Concurrency and Parallelism
Running multiple Puppeteer instances concurrently can significantly speed up the scraping process. However, this requires careful management of system resources and network connections. For example, you can launch multiple browser instances and control them using a queue system to prevent overwhelming the target website and reduce the risk of IP blocking.
Handle Network Errors in Puppeteer
Add specific error handling for network-related errors to handle potential failures during the scraping process.
page.on('requestfailed', request => {
console.error(`Request failed: ${request.url()} - ${request.failure().errorText}`);
});
Verify Proxy and API Key
Ensure your Scrape.do API key is correctly set up and that the proxy server configuration is accurate:
- Double-check that SCRAPE_DO_KEY is correctly initialized.
- Verify that your Scrape.do API key has the correct permissions for the actions you are trying to perform.
const axios = require('axios');
const puppeteer = require('puppeteer');
const fs = require('fs');
const SCRAPE_DO_TOKEN = 'YOUR_SCRAPE_DO_TOKEN'; // Replace with your actual Scrape.do token
const targetUrl = "https://66d766b074bd9da7360c1769--incandescent-gnome-f2685a.netlify.app"; // Your target URL
// Set up the request configuration for both API and Proxy modes
const isApiMode = true; // Set this to false if you want to use Proxy mode instead
// Configuration changes based on the mode
let config;
if (isApiMode) {
// API Mode: Use this configuration
const encodedUrl = encodeURIComponent(targetUrl); // URL must be encoded in API mode
config = {
method: 'GET',
url: `https://api.scrape.do?token=${SCRAPE_DO_TOKEN}&url=${encodedUrl}&super=true&geoCode=us`, // Use super=True for Residential & Mobile Proxy, set geo targeting if needed
headers: {}
};
} else {
// Proxy Mode: Use this configuration
config = {
method: 'GET',
url: targetUrl,
proxy: {
protocol: 'http', // Use either 'http' or 'https'
host: 'proxy.scrape.do',
port: 8080,
auth: {
username: SCRAPE_DO_TOKEN,
password: ''
}
}
};
}
// Fetch the page content using the configured mode
axios(config)
.then(async function (response) {
console.log('Page content fetched successfully using Scrape.do.');
//...
})
.catch(function (error) {
console.error('Error fetching page content with Scrape.do:', error.message);
});
CAPTCHA and Bot Detection
Many websites use CAPTCHA and bot detection mechanisms to prevent scraping. By integrating the Puppeteer-extra and puppeteer-extra-plugin-stealth libraries, you can disguise your headless browser as a regular user, reducing the risk of detection. Additionally, Scrape.do offers options like solve_blocking to handle more sophisticated blocking mechanisms automatically.
Dynamic Content Loading
Websites often load content dynamically through JavaScript, which can make it difficult for scrapers to capture the required data immediately. By leveraging Puppeteer’s waitForSelector, waitForFunction, or waitForTimeout methods, you can ensure that the necessary elements are fully loaded before attempting to extract data. For example, when scraping single-page applications (SPAs) that rely heavily on JavaScript, using page.waitForSelector(’.content-loaded’) ensures that content is available before proceeding.
Form Submission Timing
For web pages with dynamically loaded or asynchronously interacting elements, improper timing can cause submission errors. Use appropriate delays and wait functions, such as:
await page.waitForSelector('#open-modal'); // Wait for the button to appear
await page.click('#open-modal'); // Click the button to open the modal
await page.waitForSelector('.modal-content', { visible: true });
Handling File Uploads
To handle file uploads, use Puppeteer’s page.waitForFileChooser() and fileChooser.accept():
const [fileChooser] = await Promise.all([
page.waitForFileChooser(),
page.click('input[type="file"]'), // This will trigger the file input
]);
const filePath = path.join(os.homedir(), 'Desktop', 'test.jpg'); // Resolve the path to ~/Desktop/test.jpg
await fileChooser.accept([filePath]);
Integrating Puppeteer-extra and Puppeteer-extra-plugin-stealth
Integrating Puppeteer-extra and puppeteer-extra-plugin-stealth is crucial for bypassing common detection techniques employed by websites to identify and block scraping activities. These plugins modify the default behavior of Puppeteer, such as disabling WebGL, modifying the User-Agent string, and faking the presence of certain browser APIs that are often checked by anti-bot scripts.
To use these plugins effectively, you need to configure them properly for your specific use case. For instance, some websites check for unusual browser behaviors, such as missing fonts or disabled WebRTC. Puppeteer-extra-plugin-stealth automatically fixes these issues, making your scraping activities appear more human-like. Combining these plugins with other evasion techniques, like randomizing request intervals and using high-quality residential proxies, can significantly improve your success rate.
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
// Your scraping logic here...
})();
By integrating these plugins, your scraping scripts are less likely to be detected and blocked by anti-bot systems.
Conclusion
By using the Scrape.do API mode effectively, configuring timeouts, managing retries, and handling dynamic content, you can enhance your web scraping strategy and handle challenges such as network errors, CAPTCHAs, and bot detection. Proper configuration and strategic use of tools like Puppeteer-extra and Scrape.do features can significantly improve your scraping success rate.
- References
Also You May Interest
Ceyhun works on Web Analytics and User Behavior Analysis. He has been interested in digital technologies since 2005 and has been sharing the theoretical and practical knowledge he has gained within the framework of information ethics since 2007 through various digital channels. Information and ecology activist.