Category: Headless browser

How to Use Selenium in NodeJS

25 mins read Created Date: October 10, 2024   Updated Date: October 10, 2024

Using Selenium in NodeJS offers developers a powerful toolkit for automating web interactions, whether for testing, scraping, or simulating user behavior on dynamic websites. It lets you easily write scripts that automate complex workflows, navigate through dynamic content, and handle form submissions—tasks that are invaluable for quality assurance and data extraction.

One of the standout features of Selenium is its ability to perform cross-browser automation, allowing engineers to test or automate tasks in environments like Chrome, Firefox, and Edge. This capability ensures consistent behavior across different browsers, making Selenium an essential tool for developers seeking scalable and robust web automation solutions. In this article, we’ll tell you all you need to know about using Selenium in NodeJs.

Without further ado, let’s dive right in!

Prerequisites

Before getting started with Selenium, it’s essential to have Node.js installed on your system, as it provides the runtime environment necessary for executing JavaScript outside the browser. Selenium for NodeJS relies on modern JavaScript features and the Node Package Manager (npm), making Node.js a critical component.

You can check if Node.js is already installed by running the following command in your terminal:

node -v

This command will output the current version of Node.js. If it isn’t installed, or if your version is outdated, visitNode.js official website to download and install the latest LTS version. Alternatively, you can use a package manager like Homebrew (for macOS) or Chocolatey (for Windows) to install it. For example:

#macOS
brew install node
#WINDOWS
choco install nodejs-lts

Selenium WebDriver

The next thing you’ll be needing is Selenium WebDriver. Selenium Webdriver is essential for controlling browsers when performing automation tasks, as it acts as the interface for scripting browser interactions programmatically. For more information, refer to the official Selenium WebDriver documentation.

Browser Drivers

To perform automation, Selenium requires browser-specific drivers like ChromeDriver for Chrome or GeckoDriver for Firefox. These drivers act as the bridge between Selenium and the browser, enabling direct control over browser actions such as navigating pages, clicking elements, and filling out forms. Each driver provides the necessary instructions for Selenium to interact with the respective browser in an automated manner.

Here are some links to popular browser drivers:

It’s essential to use the driver version that is compatible with your browser version to ensure that Selenium can operate seamlessly without issues.

Installing Dependencies

To get started with Selenium and NodeJS, install a few essential packages using npm. Run the following command to install them:

npm install selenium-webdriver chromedriver

Here’s why we need each package:

  • selenium-webdriver: This package allows you to control the browser and perform automation tasks using Selenium, giving you the power to interact programmatically with web pages.
  • chromedriver: This is a browser-specific driver that connects Selenium to Chrome, enabling automation in the Chrome browser. You need to install the corresponding driver for any other browser you’d like to automate (e.g., GeckoDriver for Firefox).

These packages ensure you have the core Selenium framework and the browser driver to execute NodeJS automation scripts effectively. Now that all that is set up, let’s get to scraping!

Basic Example: Automating a Simple Task

Let’s look at a simple script demonstrating how to use Selenium in NodeJS to automate a basic task. In this example, the script will open a Chrome browser, navigate to a website, and interact with an element on the page.

// Import the selenium-webdriver library
const { Builder, By, until } = require('selenium-webdriver');

// Define an async function to execute the automation task
async function clickFirstProduct() {
  // Create a new instance of the Chrome browser
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Navigate to the e-commerce page
    await driver.get('https://www.scrapingcourse.com/ecommerce/');

    // Wait until an <li> element containing the product link with class 'woocommerce-LoopProduct-link' is available
    await driver.wait(until.elementLocated(By.css('li .woocommerce-LoopProduct-link')), 15000);

    // Find the first product link with the specified class
    let firstProduct = await driver.findElement(By.css('li .woocommerce-LoopProduct-link'));

    // Click on the first product link
    await firstProduct.click();

    // Wait for a specific element on the product page to confirm navigation
    await driver.wait(until.elementLocated(By.css('.product_title')), 15000);

    // Log that the product was successfully clicked and page loaded
    console.log("Clicked on the first product and navigated to product details page successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    // Close the browser
    await driver.quit();
  }
}

// Execute the function
clickFirstProduct();

Here’s a breakdown of key sections of this script:

  • Importing Selenium Modules: Import Builder, By, and until from selenium-webdriver for automation tasks.
  • Creating Browser Instance: Use Builder to create and configure a Chrome instance for web automation.
  • Navigating to URL: Use get() to navigate to the desired webpage.
  • Waiting for Elements: Use wait() and until.elementLocated() to wait for specific elements to be present before interacting with them.
  • Clicking Elements: Use findElement() to locate and click() to interact with elements.
  • Handling Errors: The **try-catch-finally structure manages errors and ensures proper resource cleanup.
  • Logging Actions: Use console.log() to indicate success and console.error() to report errors.
  • Closing Browser: Always use driver.quit() to close the browser once the task is complete.

Handling Asynchronous Operations in Node.js with Selenium WebDriver

Node.js is an asynchronous, event-driven runtime, meaning that operations are often non-blocking. Using NodeJs with Selenium WebDriver can create challenges, especially when dealing with dynamic content, because commands like navigating to a URL, finding elements, or interacting with a page may execute before the browser is ready.

To handle these scenarios effectively, Selenium provides mechanisms to synchronize operations, namely explicit waits and implicit waits.

Explicit waits are used to pause execution until a specific condition is met, such as the presence or visibility of an element. They provide granular control over what the script is waiting for, which is crucial when dealing with dynamically loaded content. In the above example, the wait method we used was an explicit wait.

await driver.wait(until.elementLocated(By.css('li .woocommerce-LoopProduct-link')), 15000);

Implicit waits, on the other hand, set a default waiting time for all element-finding operations in the WebDriver session. If an element is not immediately present, the WebDriver polls the DOM until it either finds the element or the specified time runs out. Here’s and example:

driver.manage().setTimeouts({ implicit: 10000 });

This tells the driver to wait up to 10 seconds before throwing a NoSuchElementException. Implicit waits are set once and apply to all element lookups, so you do not need to add waits before each element search.

When to Use Explicit Waits vs. Implicit Waits

  • Explicit Waits: Explicit waits are used when you need fine-grained control over waiting conditions. For example, if an element loads dynamically or if certain elements take longer to become visible, an explicit wait targeting that element is appropriate. Explicit waits are also preferred when interacting with AJAX content or dynamic page sections that vary in load times.

     

     

     

    await driver.wait(until.elementIsVisible(driver.findElement(By.css('.product-description'))), 10000);

    This waits until the product description becomes visible before continuing, ensuring that your script doesn’t attempt to interact with an element that isn’t ready. For pages with dynamic content that varies in load time (e.g., content loaded via AJAX), explicit waits are more reliable. They allow you to target specific conditions and provide more precise synchronization. * **Implicit Waits**: Implicit waits occur when most elements in your page take a predictable amount of time to load. They can make the script more readable and maintainable for pages that consistently render elements quickly.

driver.manage().setTimeouts({ implicit: 5000 });

This will make the script wait up to 5 seconds for any element before throwing an error. It’s suitable for simpler automation tasks with little dynamic content. If you use implicit waits, keep them short and use explicit waits where more control is needed. Implicit waits can unnecessarily slow down your script if overused, especially for already available elements.

Pro tip: Avoid combining implicit and explicit waits in the same script, as it can lead to unpredictable behavior. Use either explicit waits or implicit waits based on your use case.

Advanced Browser Interactions Using Selenium and Node.js

Selenium WebDriver is a powerful tool for automating browser interactions, allowing developers to simulate almost any user action. Beyond simply navigating to a webpage or clicking a link, there are several advanced interactions that can be useful for automating tasks on complex websites. Let’s see how to interact with dropdown menus, buttons, and alerts using Selenium in Node.js.

Interacting with Dropdown Menus in Selenium

Selenium allows you to interact with dropdowns by simulating user clicks and selecting items by their value or visible text. Here is a step-by-step example of interacting with a dropdown menu:

// Import the selenium-webdriver library
const { Builder, By, until } = require('selenium-webdriver');

async function interactWithDropdown() {
  // Step 1: Create a new instance of the Chrome WebDriver
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Step 2: Navigate to the specified e-commerce webpage
    await driver.get('example.com');

    // Step 3: Locate the dropdown menu element by its ID
    // 'dropdown-example' is assumed to be the ID of the dropdown menu on the page
    let dropdown = await driver.findElement(By.id('dropdown-example'));

    // Step 4: Wait until the dropdown is visible on the page
    // This ensures the dropdown has fully loaded and is interactable
    await driver.wait(until.elementIsVisible(dropdown), 10000); // Waits up to 10 seconds

    // Step 5: Click the dropdown menu to open it
    await dropdown.click();

    // Step 6: Wait for the dropdown options to be available
    // Here, we wait until the specific option with text 'Option 2' is located in the dropdown
    await driver.wait(until.elementLocated(By.xpath("//option[text()='Option 2']")), 10000); // Waits up to 10 seconds

    // Step 7: Locate the specific option by its visible text 'Option 2'
    let option = await driver.findElement(By.xpath("//option[text()='Option 2']"));

    // Step 8: Ensure the located option is visible before interacting with it
    await driver.wait(until.elementIsVisible(option), 10000); // Waits up to 10 seconds

    // Step 9: Click the option to select it from the dropdown menu
    await option.click();

    // Step 10: Log a message to indicate the option was successfully selected
    console.log("Selected an option from the dropdown menu.");
  } catch (error) {
    // Step 11: If an error occurs, log it to the console for debugging purposes
    console.error("An error occurred:", error);
  } finally {
    // Step 12: Quit the driver and close the browser
    // This ensures that the browser is closed properly regardless of whether an error occurred
    await driver.quit();
  }
}

// Call the function to perform the dropdown interaction
interactWithDropdown();

Pro tip: Use the visible text or value to select an option, as this approach is more readable and resilient to changes in the page structure.

Clicking Buttons with Selenium

Buttons are fundamental components in any web page, often used to submit forms or trigger JavaScript actions. Clicking buttons with Selenium is straightforward, as shown in the example below:

async function clickButton() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Navigate to the e-commerce page
    await driver.get('example.com');

    // Wait for the button to be located and visible
    let button = await driver.wait(until.elementLocated(By.css('.example-button')), 10000);

    // Click the button
    await button.click();

    console.log("Clicked the button successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    await driver.quit();
  }
}

clickButton();

Pro Tip: Always use explicit waits for interactive elements to ensure the element is ready for interaction, minimizing the risk of errors due to timing issues.

Handling JavaScript Alerts in Selenium

Alerts are small pop-up messages that can interrupt the user experience, and they can block further execution until they are dealt with, causing your script to hang or fail if ignored. These JavaScript alerts need to be explicitly handled in Selenium using the switchTo().alert() method. Here’s how you can handle alerts:

async function handleAlert() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Navigate to a page with an alert
    await driver.get('https://www.scrapingcourse.com/ecommerce/');

    // Trigger an alert (assuming there's a button that triggers an alert)
    let alertButton = await driver.findElement(By.id('alert-button'));
    await alertButton.click();

    // Wait until the alert is present
    await driver.wait(until.alertIsPresent(), 5000);

    // Switch to the alert once it appears
    let alert = await driver.switchTo().alert();

    // Get the alert text (optional)
    let alertText = await alert.getText();
    console.log("Alert says:", alertText);

    // Accept the alert
    await alert.accept();

    console.log("Alert accepted successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    await driver.quit();
  }
}

handleAlert();

Handling File Uploads with Selenium and Node.js

File uploads are a common feature in many web applications, often used for attaching documents, images, or other files. Automating this process can be very useful for testing purposes or for automating routine tasks that involve uploading files.

Selenium WebDriver allows us to handle file uploads by directly interacting with the &lt;input type="file"> element on the web page.

// Import the selenium-webdriver library
const { Builder, By, until } = require('selenium-webdriver');
const path = require('path');

async function handleFileUpload() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Navigate to the page with the file upload form
    await driver.get('https:example.com');

    // Wait for the file input element to be present
    let fileInput = await driver.wait(until.elementLocated(By.css('input[type="file"]')), 10000);

    // Define the path to the file to be uploaded
    const filePath = path.resolve(__dirname, 'sample-file.txt');

    // Set the file input value to the path of the file
    await fileInput.sendKeys(filePath);

    console.log("File path set successfully.");

    // Submit the form (optional, if a submit button is required)
    let submitButton = await driver.findElement(By.css('button[type="submit"]'));
    await submitButton.click();

    // Wait for the success message or indication (optional)
    await driver.wait(until.elementLocated(By.css('.success-message')), 5000); // Adjust selector to match the success indicator on the page
    console.log("File uploaded successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    // Close the browser
    await driver.quit();
  }
}

handleFileUpload();

The script uses Selenium WebDriver in Node.js to automate the process of uploading a file on a web page. It begins by navigating to a specified URL, waits for a file input element to become available, and then uploads a file (sample-file.txt) by sending the file path to the input element. After the file is selected, the script clicks a submit button to complete the form submission. The script also includes error handling and waits for a success message to confirm the file upload, before finally closing the browser.

Key Tips for File Uploads

  • Use Absolute Paths: The file path must be an absolute path for Selenium to work correctly. This is why we use path.resolve(), which generates a platform-independent absolute path.
  • **Interacting with the <input type="file"> Directly: Unlike other elements, you cannot use click() on the file input to open the file dialog because it requires manual user interaction. Instead, Selenium allows you to set the file path directly using sendKeys().
  • No Need for Manual Dialog Interaction: By directly providing the file path, you avoid needing to interact with the operating system’s file dialog, which isn’t possible using Selenium alone.

Practical Use Cases

  • Testing Upload Forms: Automating file uploads is critical for testing the functionality of upload forms, especially in cases where multiple files or large files need to be tested.
  • Routine Automation Tasks: Automating repetitive file uploads saves time, such as when dealing with bulk uploads for content management systems or e-commerce platforms.

Switching Between Browser Tabs or Windows Using Selenium in Node.js

When interacting with websites, you may encounter situations where new tabs or pop-up windows open, and it’s important to know how to control these different browser contexts in your NodeJs automation scripts.

Selenium WebDriver offers functionality to switch between tabs or windows using the driver.switchTo().window() method, allowing you to interact with multiple browser windows seamlessly.

// Import the selenium-webdriver library
const { Builder, By, until } = require('selenium-webdriver');

async function switchBetweenTabs() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    // Navigate to the e-commerce page
    await driver.get('https://example.com');

    // Get the current window handle (the original tab)
    let originalWindow = await driver.getWindowHandle();

    // Click a link that opens a new tab
    let link = await driver.findElement(By.css('a[target="_blank"]')); // Assuming there's a link that opens a new tab
    await link.click();

    // Wait for the new window/tab to open
    await driver.wait(async () => (await driver.getAllWindowHandles()).length === 2, 10000);

    // Get all window handles
    let allWindows = await driver.getAllWindowHandles();

    // Switch to the new window (assuming it's the second one)
    let newWindow = allWindows.find(handle => handle !== originalWindow);
    await driver.switchTo().window(newWindow);

    // Perform some actions in the new tab
    await driver.wait(until.titleIs('New Tab Title'), 10000); // Wait for the title of the new tab to be as expected
    console.log("Switched to the new tab successfully.");

    // Switch back to the original window
    await driver.switchTo().window(originalWindow);

    // Perform some actions in the original tab
    console.log("Switched back to the original tab successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    // Close the browser
    await driver.quit();
  }
}

switchBetweenTabs();

Running Tests Headlessly with Selenium in Node.js

Running tests in headless mode saves system resources and can be faster compared to running a full GUI browser. It is an ideal choice for automated testing, web scraping, and for running tests in containers or servers where a GUI isn’t available.

To run Chrome in headless mode with Selenium and Node.js, you need to set up the chromeOptions to run the browser in headless mode. Here’s how you can modify your Selenium script to do this:

// Import the selenium-webdriver and chrome modules
const { Builder, By, until } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

async function runTestsHeadlessly() {
  // Set Chrome options to run in headless mode
  let options = new chrome.Options();
  options.addArguments('--headless'); // Run Chrome in headless mode
  options.addArguments('--disable-gpu'); // Disable GPU acceleration (for compatibility)
  options.addArguments('--window-size=1920,1080'); // Set the window size (useful for responsive testing)

  // Create a new instance of the Chrome browser with the headless options
  let driver = await new Builder().forBrowser('chrome').setChromeOptions(options).build();

  try {
    // Navigate to a sample page
    await driver.get('https://www.scrapingcourse.com/ecommerce/');

    // Perform some actions (e.g., click a link)
    let link = await driver.wait(until.elementLocated(By.css('a[target="_blank"]')), 10000);
    await link.click();

    // Log a success message
    console.log("Headless test executed successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    // Close the browser
    await driver.quit();
  }
}

runTestsHeadlessly();

This script demonstrates how to use Selenium WebDriver in Node.js to perform browser automation in headless mode using Chrome.

Trade-offs of Running Tests in Headless Mode

Running tests in headless mode has several trade-offs. One major advantage is its suitability for CI/CD environments, as headless browsers don’t require a graphical interface. This makes them ideal for running tests automatically after code changes in platforms like Jenkins, GitLab CI/CD, and GitHub Actions.

Moreover, running tests without a GUI reduces resource usage, often resulting in faster execution times and increased efficiency, making it perfect for server environments and containerized applications where GUIs aren’t typically available.

However, headless mode comes with some notable disadvantages. The lack of visual feedback is a major downside, making it challenging to debug issues related to visual elements, animations, or JavaScript events, as you can’t see what the browser is doing. Furthermore, websites may render differently in headless mode compared to a GUI, leading to test failures due to subtle UI changes or timing issues.

Proper wait strategies and testing in both modes can help minimize these discrepancies. Inconsistent behavior in user interactions, such as mouse hovers or drag-and-drop actions, is another potential drawback that may require additional testing in a visible browser to ensure consistency.

To make the most of headless testing, it’s important to follow best practices. Logging actions and taking screenshots can help understand what went wrong, even when the browser isn’t visible. During development, it’s advisable to test in a regular browser to debug issues visually, then switch to headless mode for automated testing in production or CI/CD pipelines. Using explicit waits for dynamic elements ensures that elements are available and ready for interaction, compensating for the inability to observe loading and UI interactions directly.

Managing Sessions and Cookies in Selenium WebDriver: Maintaining State Across Runs

Managing cookies and sessions is essential when automating interactions with websites that require stateful behavior. For example, when logging in to a website, you might want to preserve that session across different tests or browser instances so you don’t have to log in every time. Selenium WebDriver provides functionality to add, retrieve, and delete cookies, making it easier to manage sessions and maintain state.

Let’s demonstrate how to manage cookies to maintain a session across Selenium runs. The script will:

  • Log in to a website.
  • Save the cookies to a file.
  • Load the cookies in a new session to maintain the logged-in state.

First, let’s log in to the website and save the cookies so that they can be used in future runs.

// Import the selenium-webdriver library and other required modules
const { Builder, By, until } = require('selenium-webdriver');
const fs = require('fs');
async function saveCookies() {
  let driver = await new Builder().forBrowser('chrome').build();


  try {
    // Navigate to the login page
    await driver.get('https://www.scrapingcourse.com/login');


    // Perform login (example: locate email and password fields and submit form)
    await driver.findElement(By.name('email')).sendKeys('[email protected]');
    await driver.findElement(By.name('password')).sendKeys('password');
    await driver.findElement(By.css('button[type="submit"]')).click();

    // Wait for login to complete (example: wait until the dashboard is loaded)
    await driver.wait(until.elementLocated(By.css('.catalog')), 20000);


    // Get all cookies after successful login
    let cookies = await driver.manage().getCookies();


    // Save the cookies to a file
    fs.writeFileSync('cookies.json', JSON.stringify(cookies, null, 2));
    console.log("Cookies saved successfully.");
  } catch (error) {
    console.error("An error occurred:", error);
  } finally {
    await driver.quit();
  }
}


saveCookies();

Now, we will create a new session and load the cookies saved in the previous step. This will allow us to skip the login process and go straight to an authenticated page.

const { Builder, By, until } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');
const fs = require('fs');

async function loadCookiesAndMaintainSession() {
  // Set Chrome options for better stability (including headless mode for non-GUI environments)
  let options = new chrome.Options();
  options.addArguments('--disable-gpu', '--no-sandbox', '--headless'); // Optional: use headless mode

  // Initialize the WebDriver with Chrome and options
  let driver = await new Builder()
    .forBrowser('chrome')
    .setChromeOptions(options)
    .build();

  try {
    // Step 1: Navigate to the base URL to set the cookies on the correct domain
    await driver.get('https://www.scrapingcourse.com/login');

    // Step 2: Read the saved cookies from the file
    const cookies = JSON.parse(fs.readFileSync('cookies.json'));

    // Step 3: Add each cookie to the current session, removing problematic properties like 'expiry'
    for (let cookie of cookies) {
      if ('expiry' in cookie) {
        delete cookie.expiry; // Remove 'expiry' property to avoid issues
      }
      await driver.manage().addCookie(cookie);
    }

    // Step 4: Navigate to the dashboard or any authenticated page
    await driver.get('https://www.scrapingcourse.com/dashboard');

    // Step 5: Verify that the user is logged in (wait for an element that is only visible when logged in)
    await driver.wait(until.elementLocated(By.css('.catalog')), 10000);
    console.log("Logged in using saved cookies.");

  } catch (error) {
    // Enhanced error handling
    if (error.name === 'TimeoutError') {
      console.error("Element not found. Maybe the cookies are invalid or expired.");
    } else {
      console.error("An unexpected error occurred:", error);
    }
  } finally {
    // Close the browser session
    await driver.quit();
  }
}

loadCookiesAndMaintainSession();

Common Operations for Managing Cookies

  • Get All Cookies:

    let cookies = await driver.manage().getCookies();

Retrieves all cookies associated with the current session. Add a Cookie:

await driver.manage().addCookie({ name: 'example', value: 'value123' });

Adds a cookie to the current session. This can be useful for setting custom session values.

  • Delete a Specific Cookie:

    await driver.manage().deleteCookie('example');

Delete All Cookies:

await driver.manage().deleteAllCookies();

Deletes all cookies in the current session, effectively logging the user out.

Testing and Debugging Selenium Scripts in Node.js: Logging and Debugging Best Practices

Automating browser tasks with Selenium can be highly efficient, but the more complex your interactions become, the more prone you are to encountering unexpected issues.

Proper logging and debugging techniques can save significant time and effort when troubleshooting problems in your automation scripts. Lets look at best practices for logging and debugging Selenium scripts, using both console.log() statements and screenshots to help you understand what’s happening at each step of your script.

Logging During Automation to Aid in Debugging

Logging is a powerful tool for debugging Selenium scripts. It helps you track the progress of your automation and pinpoint exactly where things go wrong. Implementing logging by adding console.log() statements at key points in your script helps you understand the flow of the script and see what steps are being executed.

const { Builder, By, until } = require('selenium-webdriver');

async function trackProgressExample() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    console.log("Starting browser and navigating to the website...");
    await driver.get('https://www.scrapingcourse.com/ecommerce/');

    console.log("Waiting for product link to be located...");
    let productLink = await driver.wait(until.elementLocated(By.css('.woocommerce-LoopProduct-link')), 10000);
    console.log("Product link found. Clicking on it...");
    await productLink.click();

    console.log("Waiting for the product details page to load...");
    await driver.wait(until.elementLocated(By.css('.product_title')), 10000);
    console.log("Product details page loaded successfully.");

  } catch (error) {
    console.error("An error occurred during the automation process:", error);
  } finally {
    console.log("Closing the browser...");
    await driver.quit();
  }
}

trackProgressExample();

Pro Tip: It’s best to log starting points, completion of important actions, and variable values to trace the progress.

Using Screenshots to Capture Browser State for Debugging

Sometimes, logging alone is not sufficient, especially if you are unsure of the browser’s state during a failure. Taking screenshots at critical points in the script helps you visually inspect what happened when something goes wrong.

const { Builder, By, until } = require('selenium-webdriver');
const fs = require('fs');

async function captureScreenshotOnError(driver, filename = 'error-screenshot.png') {
  try {
    let image = await driver.takeScreenshot();
    fs.writeFileSync(filename, image, 'base64');
    console.log(`Screenshot saved as ${filename}`);
  } catch (screenshotError) {
    console.error("Failed to capture screenshot:", screenshotError);
  }
}

async function exampleWithScreenshot() {
  let driver = await new Builder().forBrowser('chrome').build();

  try {
    console.log("Navigating to the website...");
    await driver.get('https://www.scrapingcourse.com/ecommerce/');

    console.log("Attempting to locate an element that might not exist...");
    await driver.wait(until.elementLocated(By.css('.non-existent-element')), 5000); // Will cause a timeout

  } catch (error) {
    console.error("An error occurred, capturing a screenshot:", error);
    const filename = `error-screenshot-${new Date().toISOString().replace(/[:.]/g, '-')}.png`;
    await captureScreenshotOnError(driver, filename);
  } finally {
    console.log("Closing the browser...");
    await driver.quit();
  }
}

exampleWithScreenshot();

Common Debugging Tips for Selenium Scripts

  • Increase Timeouts for Dynamic Elements: If your script often fails to find elements, it might be due to insufficient wait times. Increase explicit wait times with until.elementLocated() to ensure elements are fully loaded before interacting with them.

     

    await driver.wait(until.elementLocated(By.css('.slow-loading-element')), 15000);

Log Variables for Verification: Use console.log() to check the values of important variables, such as URLs, text extracted from elements, or responses from JavaScript actions. This helps in verifying that each step produces the expected result. ``` let pageTitle = await driver.getTitle(); console.log(“Current Page Title:”, pageTitle);

  • Check Element Visibility: Before interacting with an element, make sure it is visible. Use until.elementIsVisible() for better reliability when dealing with dynamically loaded content.

     

    let productLink = await driver.wait(until.elementLocated(By.css('.product-link')), 10000); await driver.wait(until.elementIsVisible(productLink), 5000); await productLink.click();

Use Browser Developer Tools for Troubleshooting: You can use the DevTools Console to inspect the elements and check for changes to their properties. Ensure your Selenium locators match the actual page structure. The Network tab, in particular, lets you see if there are any delays or errors in loading resources that may affect the availability of elements.

Use Pause for Debugging: Use the await driver.sleep(milliseconds) to pause execution at specific points. This is helpful when you want to observe the browser’s state manually or identify timing issues.

console.log("Pausing for 5 seconds to observe...");
await driver.sleep(5000);

Scaling Selenium Automation: Parallel Execution with Selenium Grid and Cloud Solutions

As your automation needs grow, running tests sequentially may become inefficient, especially when you need to test across multiple browsers, versions, or environments. This is where parallel execution comes into play. By running tests concurrently, you can reduce total execution time significantly, improve test coverage, and make your automation framework scalable for large-scale projects.

Running Tests in Parallel

Parallel execution allows multiple test cases to be run simultaneously, either on a single machine or distributed across multiple machines. This capability is particularly useful when:

  • Testing across multiple browser types and versions.
  • Running regression suites that contain a large number of tests.
  • Needing faster feedback loops in CI/CD pipelines.

Selenium Grid allows you to run your Selenium tests on different machines and manage multiple environments from a central hub. The hub is the central point where you load your tests, and it distributes them to different nodes, which run the tests.

  • Hub: The central server that controls all test requests. It routes commands to the appropriate nodes.
  • Node: A machine that connects to the hub and runs the browser instances as instructed by the hub.

You can check out thisquick intro on how to get started with Selenium Grid. Here’s an example that shows how to modify your script to connect to Selenium Grid:

const { Builder } = require('selenium-webdriver');

async function runTestOnBrowserStack() {
  // BrowserStack credentials from environment variables
  const USERNAME = process.env.BROWSERSTACK_USERNAME;
  const ACCESS_KEY = process.env.BROWSERSTACK_ACCESS_KEY;

  // BrowserStack Hub URL (using HTTPS for security)
  const hubUrl = `https://${USERNAME}:${ACCESS_KEY}@hub-cloud.browserstack.com/wd/hub`;

  // Define capabilities for BrowserStack
  const capabilities = {
    browserName: 'Chrome',
    browserVersion: 'latest',
    'bstack:options': {
      os: 'Windows',
      osVersion: '10',
      projectName: 'Parallel Selenium Test',
    }
  };

  let driver = await new Builder()
    .usingServer(hubUrl) // Connect to BrowserStack hub
    .withCapabilities(capabilities) // Set desired capabilities
    .build();

  try {
    // Set implicit timeout
    await driver.manage().setTimeouts({ implicit: 10000 });

    // Example automation task
    await driver.get('https://www.scrapingcourse.com/ecommerce/');
    let title = await driver.getTitle();
    console.log(`Page Title on BrowserStack: ${title}`);
  } catch (error) {
    console.error('An error occurred:', error);
  } finally {
    await driver.quit();
  }
}

// Execute the test
runTestOnBrowserStack();

Cloud Solutions for Large-Scale Automation Testing

If you need to scale your tests beyond local infrastructure or across different geographies, consider using cloud-based solutions like BrowserStack. BrowserStack is a cloud testing service that allows you to run your Selenium tests on different combinations of browsers, versions, and platforms without needing to maintain your own infrastructure.

To run your tests on BrowserStack, you need to use your BrowserStack credentials (username and access key) and specify the desired capabilities (e.g., browser type, version, OS):

const { Builder } = require('selenium-webdriver');

async function runTestOnBrowserStack() {
  // BrowserStack credentials from environment variables
  const USERNAME = process.env.BROWSERSTACK_USERNAME;
  const ACCESS_KEY = process.env.BROWSERSTACK_ACCESS_KEY;

  // BrowserStack Hub URL (using HTTPS for security)
  const hubUrl = `https://${USERNAME}:${ACCESS_KEY}@hub-cloud.browserstack.com/wd/hub`;

  // Define capabilities for BrowserStack
  const capabilities = {
    browserName: 'Chrome',
    browserVersion: 'latest',
    'bstack:options': {
      os: 'Windows',
      osVersion: '10',
      projectName: 'Parallel Selenium Test',
    }
  };

  let driver = await new Builder()
    .usingServer(hubUrl) // Connect to BrowserStack hub
    .withCapabilities(capabilities) // Set desired capabilities
    .build();

  try {
    // Set implicit timeout to handle dynamic loading
    await driver.manage().setTimeouts({ implicit: 10000 });

    // Example automation task
    await driver.get('https://www.scrapingcourse.com/ecommerce/');
    let title = await driver.getTitle();
    console.log(`Page Title on BrowserStack: ${title}`);
  } catch (error) {
    console.error('An error occurred while running the BrowserStack test:', error);
  } finally {
    // Ensure the driver quits even if an error occurs
    await driver.quit();
  }
}

// Execute the test
runTestOnBrowserStack();

Conclusion

So far, we’ve covered all you need to know to scrape data with Selenium and NodeJs. We showed how to set up your environment by installing Node.js, Selenium WebDriver, and browser drivers. Then, we used Builder() to create a browser instance and automate tasks like navigating URLs, interacting with elements, and handling asynchronous operations.

We also explored techniques for handling dynamic content, dropdowns, buttons, alerts, file uploads, and multiple tabs or windows. To optimize performance and scale automation, we considered headless mode, session management, logging, and parallel test execution.

One of the key strengths of Selenium is its ability to perform cross-browser testing, which is crucial for ensuring consistent behavior across different environments, such as Chrome, Firefox, and Edge. This is particularly important for quality assurance since it helps developers identify and fix compatibility issues before they affect users.

However, when it comes to web scraping specifically, Scrape.do is a more efficient and streamlined alternative to Selenium. Unlike Selenium, which simulates a full browser environment, making it resource-intensive and slower, Scrape.do is designed for fast and scalable scraping by providing an API-based solution. It handles challenges such as CAPTCHA-solving, IP rotation, and anti-bot measures automatically, without the need for complex setup or maintenance.

This makes Scrape.do ideal for large-scale data extraction, as it requires fewer resources, provides better reliability against scraping defenses, and eliminates the overhead of managing browser instances and compatibility issues, making it a more suitable choice for dedicated web scraping tasks. The best part? You can get started with Scrape.do now for free!

Fimber (Kasarachi) Elemuwa

Fimber (Kasarachi) Elemuwa

Senior Technical Writer

As a certified content writer and technical writer, I transform complex information into clear, concise, and user-friendly content. I have a Bachelor’s degree in Medical Microbiology and Bacteriology from the University of Port Harcourt, which gives me a solid foundation in scientific and technical writing.

Table of Contents