# How to Scrape Complete Grocery Data from Walmart with Python

> Source: https://scrape.do/blog/walmart-scraping/

Published: 2025-09-02 · Updated: 2026-08-24 · Authors: Onur Mese · Categories: Scraping Use Cases

Walmart runs America's largest grocery operation, but getting structured data from their platform is a nightmare.

The moment your scraper hits their servers, you're facing a **dual-firewall setup**: **Akamai** analyzing your TLS fingerprint while **HUMAN (PerimeterX)** watches for bot behavior patterns.

Survive both, and Walmart still won't show you real prices until you select a specific store location which affects discounts and stock availability.

In this guide, we'll show you how to extract clean, structured data from Walmart's product pages, variant listings, and category feeds without getting blocked.

[Get fully functioning code from our GitHub repo ⚙](https://github.com/scrape-do/scrapedo-scrapers/tree/main/walmart-scraper)

## Why Scraping Walmart Is Brutally Difficult

Walmart isn't like scraping a basic storefront.

It's built with layered defenses that punish bots the second they identify one.

### Akamai AND PerimeterX (HUMAN) Protection

Most retailers pick one major anti-bot vendor.

Walmart runs **both**: Akamai as a web application firewall and PerimeterX (HUMAN) as a behavioral defense layer.

***That combination is rare and extremely difficult to bypass.***

Akamai inspects every request at the TLS and header level. If your fingerprint doesn't look like a real browser, you'll be dropped with a hard block or pushed into endless redirects.

On top of that, PerimeterX waits in the background and forces a CAPTCHA challenge once it detects automation patterns:

![walmart human captcha](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/captcha_hufae6b24c09e35748b62437311c7b2ba7_12197_1200x0_resize_q80_h2_box_3.webp)

### Address / Store Selection

Even if you get through both defenses, Walmart still doesn't give you usable data.

Prices and stock are **store-specific**. A gallon of milk might be $3.98 in one store, $4.29 in another, and completely unavailable in a third.

When you first land, Walmart forces you to select a store. That choice is locked into cookies and carried through the rest of your session.

![walmart store selection bot](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/store-selection_hud65bd9df62f8b4655f0d672b5832b4b8_25094_1200x0_resize_q80_h2_box_3.webp)

Without it, every product page you scrape will either be incomplete or misleading.

### How Scrape.do Solves These

Walmart's defenses require more than headers or proxies. You need a solution that:

* Passes Akamai's TLS and header fingerprinting
* Handles HUMAN CAPTCHA challenges automatically
* Preserves cookies so a store selection sticks to your session
* Renders JavaScript for product details and dynamic pricing
* Supports referer headers for realistic pagination

***Sounds too good to be in just one solution, right?***

But it's real. Scrape.do provides all of these in one request layer.

By using parameters like `super=true` (browser fingerprinting), `render=true` (JavaScript rendering), and `extraHeaders=true` (custom cookies and referers), the scraper behaves like a real customer session tied to a valid store.

> **⭐ Even better**: the [Walmart Scraper API](/products/ready-api/walmart-scraper/) handles store selection automatically. Instead of manually extracting cookies, you can simply pass a zip code and store ID to lock your session to any Walmart location instantly. It also has structured `/search` and `/product` endpoints that return JSON instead of HTML. See the [Scrape.do Walmart Scraper Docs](/documentation/walmart-scraper-api/), or skip to [Or The Easy Way](#or-the-easy-way) below.

## Setting Up an Address

Walmart doesn't return correct prices or stock data until your session is tied to a store. That tie happens through cookies.

We'll explore two different methods to handle store selection:

### Method 1: Manual Cookie Extraction

When you pick a store in the browser, Walmart sets several cookies on your session.

Three are essential:

* **ACID**
* **locDataV3**
* **locGuestData**

Without these, product pages fall back to default data or show incomplete inventory.

#### Finding the Cookies in Your Browser

Open Walmart in Chrome, select a store from the top menu, then open **Developer Tools → Application → Storage → Cookies**. You'll see all active cookies for `walmart.com`.

The three we care about are highlighted here:

![walmart cookie scraping](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-cookie-scraping_hu71c8f4814f281b1a20060032325527d3_67420_1200x0_resize_q80_h2_box_3.webp)

Each cookie has a long opaque value. Copy them exactly as-is, no spaces or line breaks.

#### Using Cookies in Requests

We'll send them via the `sd-Cookie` header when making Scrape.do requests. This is how Walmart knows your session is already locked to a store.

Example configuration:

```python
headers = {
    "sd-Cookie": "ACID=...; locDataV3=...; locGuestData=...",
    "sd-User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)..."
}
```

Attach these headers in every request you send to Scrape.do. If the cookies expire or if you want to get cookies for a different store, grab a new set from DevTools and replace them.

### Method 2: Scrape.do Walmart Plugin (Simple)

Scrape.do makes store selection much easier with its new Walmart plugin.

Instead of manually extracting cookies, you can set a store location with just two parameters: `zipcode` and `storeid`.

Here's how the plugin works:

**Regular Scrape.do API call:**

```python
api_url = f"https://api.scrape.do/?token={TOKEN}&url={encoded_url}&super=true&geoCode=us&render=true"
```

**With Walmart plugin and a store selected:**

```python
api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}"
```

The plugin automatically handles all the cookie management, session persistence, and store selection that you'd normally have to do manually.

#### Finding Your Store ID

To use the plugin, you need two pieces of information: your zip code (easy) and the store ID (requires a quick lookup).

Here's how to find the store ID:

1. Visit Walmart.com and click the "Pickup or delivery?" dropdown

   ![walmart store selection](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-store-id-select_hu900465a642897f6b4d77df07a48b2a39_67799_1200x0_resize_q80_h2_box_3.webp)

2. Enter your zip code and select your preferred store
3. Click on "Store details" for your selected store

   ![walmart store details](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-store-id-select-details_hu0a118f093117227534bf811463b5288e_24435_1200x0_resize_q80_h2_box_3.webp)

4. This takes you to a URL like `https://www.walmart.com/store/3520-secaucus-nj`
5. The store ID is the number at the beginning: `3520`

For our examples, we'll use the Secaucus Supercenter (Store ID: 3520, Zip: 07094).

**The plugin approach saves you from:**

* Manually extracting cookies from DevTools
* Managing cookie expiration
* Handling session persistence
* Dealing with store-specific cookie chains

We'll use the Scrape.do Walmart plugin for all our code examples going forward.

## Scrape Individual Product Pages

Now we'll extract data from single products, including name, brand, price, stock state, discounts, and details.

We'll cover all price variations you might encounter too, whether **in stock** at regular price, **out of stock,** or with **discounts applied.**

### Setup and Prerequisites

Start with the essential imports and configuration:

We'll need `requests` for HTTP calls, `BeautifulSoup` for HTML parsing, and `urllib.parse` for URL encoding. We'll use Scrape.do's Walmart plugin to automatically handle store selection.

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup

# Scrape.do API token
TOKEN = "<your-token>"

# Target Walmart product URL
target_url = "https://www.walmart.com/ip/17221971947"
encoded_url = urllib.parse.quote(target_url)

# Scrape.do Walmart plugin API endpoint with Secaucus Supercenter
api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"

response = requests.get(api_url)
```

### Scrape Product Name and Brand

We'll parse the HTML response with BeautifulSoup. The product name is usually inside an `h1` element with `itemprop="name"`.

For the brand, Walmart doesn't always use a consistent structure, so we check two possible locations:

1. An anchor tag with the class `prod-brandName`
2. An anchor tag with the attribute `data-seo-id="brand-name"`

By checking both, we reduce the chance of missing brand information when Walmart changes its frontend.

```python

if response.status_code != requests.codes.ok:
    response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

# Product name
name_tag = soup.find("h1", {"itemprop": "name"})
name = name_tag.get_text(strip=True) if name_tag else "N/A"

# Brand (two different possible selectors)
brand_tag = soup.find("a", {"class": "prod-brandName"})
if not brand_tag:
    brand_tag = soup.find("a", {"data-seo-id": "brand-name"})
brand = brand_tag.get_text(strip=True) if brand_tag else "N/A"
```

### Scrape Stock Availability, Product Price and Discount

Walmart product pages expose stock, pricing, and discount information in multiple elements depending on context.

To handle this, we'll target the most stable attributes available.

* **Price**: usually found inside a `span` with `data-seo-id="hero-price"`. If the text includes "Now", we split and take the second dollar value to isolate the actual price.
* **Stock availability**: if the "Add to Cart" button exists (`button` with `data-dca-name="ItemBuyBoxAddToCartButton"`), the item is in stock. If not, we mark it as out of stock.
* **Discount**: when a product is on sale, a `div` with `data-testid="dollar-saving"` shows the savings amount.

```python
# Price
price_tag = soup.find("span", {"data-seo-id": "hero-price"})
if price_tag:
    price = price_tag.get_text(strip=True)
    if "Now" in price:
        price = "$" + price.split("$")[1]
else:
    price = "N/A"

# Stock availability
stock = soup.find("button", {"data-dca-name": "ItemBuyBoxAddToCartButton"})
stock = "In stock" if stock else "Out of stock"

# Discount
discount = soup.find("div", {"data-testid": "dollar-saving"})
discount = "$" + discount.get_text(strip=True).split("$")[1] if discount else "N/A"
```

### Scrape Product Details

Walmart product pages may expose details in two different formats.

Some products use **AI-generated summaries**, while others use the **traditional description list**. We'll handle both of these cases:

1. `div#product-smart-summary div.mt0 li` → usually AI-generated details
2. `span#product-description-atf li` → fallback for regular details

We'll also capture the main product image. This is stored in an `img` tag with the attribute `data-seo-id="hero-image"`.

```python
additional_info = []

# First attempt: AI generated details
for li in soup.select("div#product-smart-summary div.mt0 li"):
    additional_info.append(li.get_text(strip=True))

# Fallback: regular details
if not len(additional_info):
    for li in soup.select("span#product-description-atf li"):
        additional_info.append(li.get_text(strip=True))

# Image URL
img_tag = soup.find("img", {"data-seo-id": "hero-image"})
image = img_tag["src"] if img_tag and img_tag.has_attr("src") else "N/A"
```

### Full Code for Single Product Scraping

Here's the complete working script.

It establishes a store-locked session, fetches the product page through Scrape.do's proxy network, and extracts all available product data.

The output shows structured information ready for analysis or storage.

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup

# Scrape.do API token
TOKEN = "<your-token>"

# Target Walmart product URL
target_url = "https://www.walmart.com/ip/17221971947"
encoded_url = urllib.parse.quote(target_url)

# Scrape.do Walmart plugin API endpoint with Secaucus Supercenter
api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"

response = requests.get(api_url)
if response.status_code != requests.codes.ok:
    response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

# Product name
name_tag = soup.find("h1", {"itemprop": "name"})
name = name_tag.get_text(strip=True) if name_tag else "N/A"

# Brand (two different possible selectors)
brand_tag = soup.find("a", {"class": "prod-brandName"})
if not brand_tag:
    brand_tag = soup.find("a", {"data-seo-id": "brand-name"})
brand = brand_tag.get_text(strip=True) if brand_tag else "N/A"

# Price
price_tag = soup.find("span", {"data-seo-id": "hero-price"})
if price_tag:
    price = price_tag.get_text(strip=True)
    if "Now" in price:
        price = "$" + price.split("$")[1]
else:
    price = "N/A"

# Stock availability
stock = soup.find("button", {"data-dca-name": "ItemBuyBoxAddToCartButton"})
stock = "In stock" if stock else "Out of stock"

# Discount
discount = soup.find("div", {"data-testid": "dollar-saving"})
discount = "$" + discount.get_text(strip=True).split("$")[1] if discount else "N/A"

# Additional product details (AI or regular)
additional_info = []
for li in soup.select("div#product-smart-summary div.mt0 li"):
    additional_info.append(li.get_text(strip=True))
if not len(additional_info):
    for li in soup.select("span#product-description-atf li"):
        additional_info.append(li.get_text(strip=True))

# Image URL
img_tag = soup.find("img", {"data-seo-id": "hero-image"})
image = img_tag["src"] if img_tag and img_tag.has_attr("src") else "N/A"

# Print results
print("Product Name:", name)
print("Brand:", brand)
print("Price:", price)
print("Stock:", stock)
print("Discount:", discount)
print("Additional Info:", additional_info)
print("Image:", image)
```

When you run the script, it prints structured information for the target product. Here's a sample console output:

![scrape single products from walmart](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/single-product-scraping-walmart_hu2d0602fd93a3333125c68cffceaad42d_19551_1200x0_resize_q80_h2_box_3.webp)

## Scrape Products with Variants

Many Walmart items aren't static.

A single listing can contain multiple **variants** such as different sizes, pack counts, or colors.

Each variant may have its own price, stock state, and discount, which means scraping only the base product page will miss most of the catalog.

To handle this, we'll collect all variant URLs from the product page, then loop through them one by one to capture details. We'll save the output in a CSV where each row represents one variant.

![walmart item variants](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-item-variants-scraping_hu19eec2fe9ab0d4ec32c46d08da5d9033_44728_1200x0_resize_q80_h2_box_3.webp)

### Setup and Prerequisites

Variant scraping requires the same foundation as single products, but we'll add CSV handling since we're collecting multiple items.

The Scrape.do plugin ensures each variant shows pricing for your selected store location.

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup
import csv

# Scrape.do API token
TOKEN = "<your-token>"

# Walmart product page with variants
target_url = "https://www.walmart.com/ip/17221971947"
encoded_url = urllib.parse.quote(target_url)

# Scrape.do Walmart plugin API endpoint with Secaucus Supercenter
api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"
```

### Scraping Variant URLs

Walmart lists different options for a product (colors, pack sizes, etc.) inside a variant group container hidden within the HTML.

Each variant is represented by an anchor element within `div#item-page-variant-group-bg-div`:

![walmart item variants html](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-item-variants-html_hu3611080475129cc12b4bac91ad658d1a_55145_1200x0_resize_q80_h2_box_3.webp)

We'll extract every link, append to Walmart's base domain, and then remove duplicates to avoid hitting the same variant twice.

```python
response = requests.get(api_url)
if response.status_code != requests.codes.ok:
    response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

variant_urls = []
for variant in soup.select("div#item-page-variant-group-bg-div a"):
    variant_urls.append("https://www.walmart.com/" + variant.get("href"))

# Remove duplicates
variant_urls = list(set(variant_urls))
```

### Looping Through Variants and Parsing Data

With the list of variant URLs ready, we'll fetch each page through Scrape.do and extract the same fields as in the single product scraper.

```python
variant_products = []

for url in variant_urls:
    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"

    response = requests.get(api_url)
    if response.status_code != requests.codes.ok:
        response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Product name
    name_tag = soup.find("h1", {"itemprop": "name"})
    name = name_tag.get_text(strip=True) if name_tag else "N/A"

    # Brand
    brand_tag = soup.find("a", {"class": "prod-brandName"})
    if not brand_tag:
        brand_tag = soup.find("a", {"data-seo-id": "brand-name"})
    brand = brand_tag.get_text(strip=True) if brand_tag else "N/A"

    # Price
    price_tag = soup.find("span", {"data-seo-id": "hero-price"})
    if price_tag:
        price = price_tag.get_text(strip=True)
        if "Now" in price:
            price = "$" + price.split("$")[1]
    else:
        price = "N/A"

    # Stock
    stock = soup.find("button", {"data-dca-name": "ItemBuyBoxAddToCartButton"})
    stock = "In stock" if stock else "Out of stock"

    # Discount
    discount = soup.find("div", {"data-testid": "dollar-saving"})
    discount = "$" + discount.get_text(strip=True).split("$")[1] if discount else "N/A"

    # Additional product details
    additional_info = []
    for li in soup.select("div#product-smart-summary div.mt0 li"):
        additional_info.append(li.get_text(strip=True))
    if not len(additional_info):
        for li in soup.select("span#product-description-atf li"):
            additional_info.append(li.get_text(strip=True))

    # Image
    img_tag = soup.find("img", {"data-seo-id": "hero-image"})
    image = img_tag["src"] if img_tag and img_tag.has_attr("src") else "N/A"

    variant_products.append({
        "Product Name": name,
        "Brand": brand,
        "Price": price,
        "Stock": stock,
        "Discount": discount,
        "Additional Info": additional_info,
        "Image": image
    })
```

This gives us a list of dictionaries stored in the **variant\_products** variable, each representing a specific product variation with its full details.

### Finalize and Export

Here's the complete script that ties everything together.

> 💡 For the export step we'll use Python's built-in csv library to write each variant as a row in the output file.

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup
import csv

# Scrape.do API token
TOKEN = "<your-token>"

# Walmart product page with variants
target_url = "https://www.walmart.com/ip/17221971947"
encoded_url = urllib.parse.quote(target_url)

# Scrape.do Walmart plugin API endpoint with Secaucus Supercenter
api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"

# Step 1: collect variant URLs
response = requests.get(api_url)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

variant_urls = []
for variant in soup.select("div#item-page-variant-group-bg-div a"):
    variant_urls.append("https://www.walmart.com/" + variant.get("href"))
variant_urls = list(set(variant_urls))

# Step 2: loop through variants and extract data
variant_products = []

for url in variant_urls:
    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode=07094&storeid=3520&url={encoded_url}&super=true&geoCode=us&render=true"

    response = requests.get(api_url)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    name_tag = soup.find("h1", {"itemprop": "name"})
    name = name_tag.get_text(strip=True) if name_tag else "N/A"

    brand_tag = soup.find("a", {"class": "prod-brandName"})
    if not brand_tag:
        brand_tag = soup.find("a", {"data-seo-id": "brand-name"})
    brand = brand_tag.get_text(strip=True) if brand_tag else "N/A"

    price_tag = soup.find("span", {"data-seo-id": "hero-price"})
    if price_tag:
        price = price_tag.get_text(strip=True)
        if "Now" in price:
            price = "$" + price.split("$")[1]
    else:
        price = "N/A"

    stock = soup.find("button", {"data-dca-name": "ItemBuyBoxAddToCartButton"})
    stock = "In stock" if stock else "Out of stock"

    discount = soup.find("div", {"data-testid": "dollar-saving"})
    discount = "$" + discount.get_text(strip=True).split("$")[1] if discount else "N/A"

    additional_info = []
    for li in soup.select("div#product-smart-summary div.mt0 li"):
        additional_info.append(li.get_text(strip=True))
    if not len(additional_info):
        for li in soup.select("span#product-description-atf li"):
            additional_info.append(li.get_text(strip=True))

    img_tag = soup.find("img", {"data-seo-id": "hero-image"})
    image = img_tag["src"] if img_tag and img_tag.has_attr("src") else "N/A"

    variant_products.append({
        "Product Name": name,
        "Brand": brand,
        "Price": price,
        "Stock": stock,
        "Discount": discount,
        "Additional Info": additional_info,
        "Image": image
    })

# Step 3: save results to CSV
with open("variant_products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, variant_products[0].keys())
    writer.writeheader()
    writer.writerows(variant_products)
```

*And here's what our output will look like:*

![extracting variant item data from walmart](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/walmart-item-variants-extracted_huee5aa5e6b32847f7071ccb25e82a776d_52483_1200x0_resize_q80_h2_box_3.webp)

## Scrape Categories with Multiple Pages

Individual product pages only show you one item at a time.

To collect broader datasets, you need to scrape Walmart's category listings, which can span dozens of paginated result pages.

Each page contains tiles with product name, price, image, review data, and a link to the product page.

We need to handle these two challenges:

* Pagination is handled through URL parameters (`?page=2`, `?page=3`, etc.), so we need to loop through them in sequence.
* Walmart often reuses the same product tiles across pages, so we'll also track duplicates to avoid cluttering our CSV with repeated items.

We'll use helper functions to **fetch pages through Scrape.do**, **parse products into structured fields**, and **loop through all pages** until we reach the end.

![walmart product tiles](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/product-tile_huc0a6eb0952fcba8bbd3d8cafe9a18130_637145_1200x0_resize_q80_h2_box_3.webp)

### Setup and Prerequisites

When scraping Walmart categories, pagination can trigger bot defenses.

If you try to hit `?page=2` directly without context, Walmart treats it as suspicious traffic. The workaround is to always send a **referer header** that points to the previous page.

Instead of hardcoding it each time, we'll build helper functions that attach the correct referer for us:

* Page 1 refers to the base URL.
* Page 2 refers to page 1.
* Page 3 refers to page 2, and so on.

Here's the setup section of the code:

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup
import csv
import time

# Scrape.do token
TOKEN = "<your-token>"

# Walmart category page (example: deli meats & cheeses)
target_url = "https://www.walmart.com/browse/food/shop-all-deli-sliced-meats-cheeses/976759_976789_5428795_9084734"

# Store info for Secaucus Supercenter
ZIPCODE = "07094"
STORE_ID = "3520"
```

### Building Necessary Helpers

We split the logic into two helpers so the main scraper stays readable.

#### fetch\_page(url, previous\_url)

* Sends the request through Scrape.do's Walmart plugin
* Automatically handles store selection with zip code and store ID
* Adds referer header to maintain realistic navigation patterns
* Handles URL encoding and returns raw HTML

```python
def fetch_page(url, previous_url):
    headers = {
        "sd-User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
        "sd-Referer": previous_url
    }

    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode={ZIPCODE}&storeid={STORE_ID}&url={encoded_url}&super=true&geoCode=us&extraHeaders=true&render=true&customWait=5000&blockResources=false"

    response = requests.get(api_url, headers=headers)
    response.raise_for_status()
    return response.text
```

> We're adding some extra Scrape.do parameters to ensure that all the tile items are loaded correctly.

#### parse\_products(html)

* Parses each product tile in the category grid
* Extracts:
  * Name and link
  * Image URL
  * Price
  * Review count and rating
* Returns a list of product dictionaries

```python
def parse_products(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []

    for tile in soup.select("[data-dca-name='ui_product_tile:vertical_index']"):
        product = {}

        # Name
        name_tag = tile.select_one("span", {"data-automation-id": "product-title"}).get_text(strip=True)
        product["Name"] = name_tag.split("$")[0] if "$" in name_tag else name_tag

        # Link
        link_tag = tile.select_one("a", href=True)
        if link_tag:
            href = link_tag["href"]
            if href.startswith("/"):
                href = "https://www.walmart.com" + href
            product["Link"] = href
        else:
            product["Link"] = ""

        # Image
        img_tag = tile.select_one("img", {"data-testid": "productTileImage"})
        product["Image"] = img_tag["src"] if img_tag and img_tag.has_attr("src") else ""

        # Price
        price_tag = tile.find("div", {"data-automation-id": "product-price"})
        price_section = price_tag.find("span", {"class": "w_iUH7"}) if price_tag else None
        product["Price"] = "$" + price_section.get_text(strip=True).split("$")[1].split(" ")[0] if price_section else ""

        # Reviews & rating
        review_count_tag = tile.select_one('span[data-testid="product-reviews"]')
        product["ReviewCount"] = review_count_tag.get("data-value") if review_count_tag else None

        product["Rating"] = ""
        if review_count_tag:
            next_span = review_count_tag.find_next_sibling("span")
            if next_span:
                product["Rating"] = next_span.get_text(strip=True).split(" ")[0]

        products.append(product)

    return products
```

### Looping Through Multiple Pages

While creating our looping logic we'll also add two safeguards:

* **Stop early** if a page returns no products.
* **Track duplicates** using product image URLs so the same product isn't stored twice across pages.

```python
def scrape_category(base_url, max_pages=50, delay=1.5):
    all_products = []
    seen_images = set()  # track unique product images across pages
    page = 1

    while page <= max_pages:
        if page == 1:
            html = fetch_page(base_url, base_url)
        elif page == 2:
            html = fetch_page(f"{base_url}?page=2", base_url)
        else:
            html = fetch_page(f"{base_url}?page={page}", f"{base_url}?page={page-1}")

        products = parse_products(html)
        if not products:
            print(f"No products found on page {page}. Stopping.")
            break

        new_count = 0
        for prod in products:
            img = prod.get("Image")
            if img and img not in seen_images:
                seen_images.add(img)
                all_products.append(prod)
                new_count += 1
            else:
                print(f"Duplicate skipped: {prod.get('Name')}")

        print(f"Page {page}: extracted {new_count} new products (total {len(all_products)})")

        page += 1
        time.sleep(delay)  # polite delay
    return all_products
```

### Full Code and CSV Export

Here's the complete script that brings together our setup, helpers, and pagination loop.

It scrapes every page in the target category, removes duplicate products across pages.

We'll add CSV export at the end using Python's built-in csv library as we did in previous sections:

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup
import csv
import time

# Scrape.do token
TOKEN = "<your-token>"

# Walmart category page (example: deli meats & cheeses)
target_url = "https://www.walmart.com/browse/food/shop-all-deli-sliced-meats-cheeses/976759_976789_5428795_9084734"

# Store info for Secaucus Supercenter
ZIPCODE = "07094"
STORE_ID = "3520"

def fetch_page(url, previous_url):
    headers = {
        "sd-User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
        "sd-Referer": previous_url
    }

    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode={ZIPCODE}&storeid={STORE_ID}&url={encoded_url}&super=true&geoCode=us&extraHeaders=true&render=true&customWait=5000&blockResources=false"

    response = requests.get(api_url, headers=headers)
    response.raise_for_status()
    return response.text

def parse_products(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []

    for tile in soup.select("[data-dca-name='ui_product_tile:vertical_index']"):
        product = {}

        # Name
        name_tag = tile.select_one("span", {"data-automation-id": "product-title"}).get_text(strip=True)
        product["Name"] = name_tag.split("$")[0] if "$" in name_tag else name_tag

        # Link
        link_tag = tile.select_one("a", href=True)
        if link_tag:
            href = link_tag["href"]
            if href.startswith("/"):
                href = "https://www.walmart.com" + href
            product["Link"] = href
        else:
            product["Link"] = ""

        # Image
        img_tag = tile.select_one("img", {"data-testid": "productTileImage"})
        product["Image"] = img_tag["src"] if img_tag and img_tag.has_attr("src") else ""

        # Price
        price_tag = tile.find("div", {"data-automation-id": "product-price"})
        price_section = price_tag.find("span", {"class": "w_iUH7"}) if price_tag else None
        product["Price"] = "$" + price_section.get_text(strip=True).split("$")[1].split(" ")[0] if price_section else ""

        # Reviews & rating
        review_count_tag = tile.select_one('span[data-testid="product-reviews"]')
        product["ReviewCount"] = review_count_tag.get("data-value") if review_count_tag else None

        product["Rating"] = ""
        if review_count_tag:
            next_span = review_count_tag.find_next_sibling("span")
            if next_span:
                product["Rating"] = next_span.get_text(strip=True).split(" ")[0]

        products.append(product)

    return products

def scrape_category(base_url, max_pages=50, delay=1.5):
    all_products = []
    seen_images = set()  # track unique product images across pages
    page = 1

    while page <= max_pages:
        if page == 1:
            html = fetch_page(base_url, base_url)
        elif page == 2:
            html = fetch_page(f"{base_url}?page=2", base_url)
        else:
            html = fetch_page(f"{base_url}?page={page}", f"{base_url}?page={page-1}")

        products = parse_products(html)
        if not products:
            print(f"No products found on page {page}. Stopping.")
            break

        new_count = 0
        for prod in products:
            img = prod.get("Image")
            if img and img not in seen_images:
                seen_images.add(img)
                all_products.append(prod)
                new_count += 1
            else:
                print(f"Duplicate skipped: {prod.get('Name')}")

        print(f"Page {page}: extracted {new_count} new products (total {len(all_products)})")

        page += 1
        time.sleep(delay)  # polite delay
    return all_products

# Run scraper and save output
rows = scrape_category(target_url, max_pages=20)
with open("walmart_category.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["Name", "Price", "Link", "Image", "Rating", "ReviewCount"])
    writer.writeheader()
    writer.writerows(rows)

print(f"Extracted {len(rows)} products to walmart_category.csv")
```

This will take a while but successfully extract all items from the category **without duplicates:**

![scrape walmart categories](/uploads/blog/how-to-scrape-complete-grocery-data-from-walmart-with-python/category-output_hu7832375eec71fcdc7f8c4c7348ea4401_98637_1200x0_resize_q80_h2_box_3.webp)

## Use Case: Track Stock and Price Changes on Walmart

Scraping single products and categories is useful, but the real power comes from tracking changes over time.

Price drops, stock outages, and restocks are all events worth detecting automatically.

Imagine getting an alert when a product's price is reduced, when stock suddenly disappears, or when an item comes back in stock. For larger projects, multiple category scrapers can also feed into a central dataset to build a long-term history of Walmart's inventory.

We'll extend our approach into a tracker script that:

1. Collects category results into a structured dataset
2. Saves them into a snapshot file
3. Compares today's results with the previous run
4. Prints alerts when stock or price changes occur

### Building Necessary Helpers

We'll define three helpers to keep the tracking logic modular.

#### fetch\_page(url, previous\_url)

* Sends requests through Scrape.do's Walmart plugin
* Automatically handles store selection and session management
* Adds referer header so pagination looks natural
* Returns raw HTML for parsing

```python
def fetch_page(url, previous_url):
    headers = {
        "sd-User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
        "sd-Referer": previous_url
    }
    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode={ZIPCODE}&storeid={STORE_ID}&url={encoded_url}&super=true&geoCode=us&extraHeaders=true&render=true&customWait=5000&blockResources=false"
    response = requests.get(api_url, headers=headers)
    response.raise_for_status()
    return response.text
```

#### parse\_products(html)

* Parses each product tile in the category grid
* Extracts name, link, image, price, rating, and review count
* Marks **stock availability** based on the presence of a button with `data-automation-id="add-to-cart"`

```python
def parse_products(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for tile in soup.select("[data-dca-name='ui_product_tile:vertical_index']"):
        product = {}

        # Name
        name_tag = tile.select_one("span", {"data-automation-id": "product-title"})
        product["Name"] = name_tag.get_text(strip=True) if name_tag else "N/A"

        # Link
        link_tag = tile.select_one("a", href=True)
        if link_tag:
            href = link_tag["href"]
            if href.startswith("/"):
                href = "https://www.walmart.com" + href
            product["Link"] = href
        else:
            product["Link"] = ""

        # Image
        img_tag = tile.select_one("img", {"data-testid": "productTileImage"})
        product["Image"] = img_tag["src"] if img_tag and img_tag.has_attr("src") else ""

        # Price
        price_tag = tile.find("div", {"data-automation-id": "product-price"})
        price_section = price_tag.find("span", {"class": "w_iUH7"}) if price_tag else None
        product["Price"] = price_section.get_text(strip=True) if price_section else "N/A"

        # Reviews & Rating
        review_count_tag = tile.select_one('span[data-testid="product-reviews"]')
        product["ReviewCount"] = review_count_tag.get("data-value") if review_count_tag else None
        product["Rating"] = ""
        if review_count_tag:
            next_span = review_count_tag.find_next_sibling("span")
            if next_span:
                product["Rating"] = next_span.get_text(strip=True).split(" ")[0]

        # Stock availability
        add_to_cart = tile.find("button", {"data-automation-id": "add-to-cart"})
        product["Stock"] = "In stock" if add_to_cart else "Out of stock"

        products.append(product)
    return products
```

#### Snapshot Helpers

* **`save_snapshot(filename, rows)`** writes current run results to a JSON file.
* **`load_snapshot(filename)`** loads the last run for comparison, returns empty list if none exists.

```python
def save_snapshot(filename, rows):
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(rows, f, ensure_ascii=False, indent=2)

def load_snapshot(filename):
    if os.path.exists(filename):
        with open(filename, "r", encoding="utf-8") as f:
            return json.load(f)
    return []
```

### Looping Through Categories and Comparing Results

We'll store each snapshot as a JSON file named after the category.

On each run, we'll load previous run's snapshot, scrape the current one, then print any differences.

```python
def scrape_category(base_url, max_pages=5, delay=1.5):
    all_products = []
    page = 1
    while page <= max_pages:
        if page == 1:
            html = fetch_page(base_url, base_url)
        elif page == 2:
            html = fetch_page(f"{base_url}?page=2", base_url)
        else:
            html = fetch_page(f"{base_url}?page={page}", f"{base_url}?page={page-1}")

        products = parse_products(html)
        if not products:
            break

        all_products.extend(products)
        page += 1
        time.sleep(delay)
    return all_products


def compare_snapshots(old_data, new_data):
    alerts = []

    # Build quick lookup by product link
    old_map = {p["Link"]: p for p in old_data}
    for prod in new_data:
        link = prod["Link"]
        old_prod = old_map.get(link)

        if old_prod:
            # Check price change
            if prod["Price"] != old_prod.get("Price"):
                alerts.append(f"Price change for {prod['Name']}: {old_prod.get('Price')} -> {prod['Price']}")

            # Check stock change
            if prod["Stock"] != old_prod.get("Stock"):
                alerts.append(f"Stock change for {prod['Name']}: {old_prod.get('Stock')} -> {prod['Stock']}")

    return alerts
```

### Full Code for Tracking Stock and Price Changes

This script scrapes multiple categories, saves each run as a JSON snapshot, compares it with the last run, and exports the latest data into CSV.

Alerts are printed in the console when a product's price or stock state changes.

```python
import requests
import urllib.parse
from bs4 import BeautifulSoup
import csv
import time
import json
import os

# Scrape.do token
TOKEN = "<your-token>"

# Walmart categories to track
CATEGORY_URLS = [
    "https://www.walmart.com/browse/food/shop-all-deli-sliced-meats-cheeses/976759_976789_5428795_9084734",
    "https://www.walmart.com/browse/food/snacks/976759_976787"
]

# Store info for Secaucus Supercenter
ZIPCODE = "07094"
STORE_ID = "3520"

def fetch_page(url, previous_url):
    headers = {
        "sd-User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
        "sd-Referer": previous_url
    }

    encoded_url = urllib.parse.quote(url)
    api_url = f"https://api.scrape.do/plugin/walmart/store?token={TOKEN}&zipcode={ZIPCODE}&storeid={STORE_ID}&url={encoded_url}&super=true&geoCode=us&extraHeaders=true&render=true&customWait=5000&blockResources=false"

    response = requests.get(api_url, headers=headers)
    response.raise_for_status()
    return response.text

def parse_products(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for tile in soup.select("[data-dca-name='ui_product_tile:vertical_index']"):
        product = {}

        # Name
        name_tag = tile.select_one("span[data-automation-id='product-title']")
        product["Name"] = name_tag.get_text(strip=True) if name_tag else "N/A"

        # Link
        link_tag = tile.select_one("a[href]")
        href = link_tag["href"] if link_tag else ""
        href = "https://www.walmart.com" + href if href.startswith("/") else href
        product["Link"] = href

        # Image
        img_tag = tile.select_one("img[data-testid='productTileImage']")
        product["Image"] = img_tag["src"] if img_tag and img_tag.has_attr("src") else ""

        # Price
        price_tag = tile.find("div", {"data-automation-id": "product-price"})
        price_section = price_tag.find("span", {"class": "w_iUH7"}) if price_tag else None
        product["Price"] = price_section.get_text(strip=True) if price_section else "N/A"

        # Reviews & Rating
        review_count_tag = tile.select_one('span[data-testid="product-reviews"]')
        product["ReviewCount"] = review_count_tag.get("data-value") if review_count_tag else None
        product["Rating"] = ""
        next_span = review_count_tag.find_next_sibling("span") if review_count_tag else None
        product["Rating"] = next_span.get_text(strip=True).split(" ")[0] if next_span else ""

        # Stock availability
        add_to_cart = tile.find("button", {"data-automation-id": "add-to-cart"})
        product["Stock"] = "In stock" if add_to_cart else "Out of stock"

        products.append(product)
    return products

def scrape_category(base_url, max_pages=5, delay=1.5):
    all_products = []
    page = 1
    while page <= max_pages:
        url = base_url if page == 1 else f"{base_url}?page={page}"
        previous_url = base_url if page <= 2 else f"{base_url}?page={page-1}"

        html = fetch_page(url, previous_url)
        products = parse_products(html)

        break if not products else None

        all_products.extend(products)
        page += 1
        time.sleep(delay)
    return all_products

def save_snapshot(filename, rows):
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(rows, f, ensure_ascii=False, indent=2)

def load_snapshot(filename):
    return json.load(open(filename, "r", encoding="utf-8")) if os.path.exists(filename) else []

def compare_snapshots(old_data, new_data):
    alerts = []
    old_map = {p["Link"]: p for p in old_data}
    for prod in new_data:
        link = prod["Link"]
        old_prod = old_map.get(link)
        alerts.append(f"Price change for {prod['Name']}: {old_prod.get('Price')} -> {prod['Price']}") if old_prod and prod["Price"] != old_prod.get("Price") else None
        alerts.append(f"Stock change for {prod['Name']}: {old_prod.get('Stock')} -> {prod['Stock']}") if old_prod and prod["Stock"] != old_prod.get("Stock") else None
    return alerts

# Main tracking logic
for url in CATEGORY_URLS:
    category_name = url.split("/")[5]  # crude way to name file
    snapshot_file = f"{category_name}_snapshot.json"

    old_snapshot = load_snapshot(snapshot_file)
    new_snapshot = scrape_category(url, max_pages=20)
    save_snapshot(snapshot_file, new_snapshot)

    # Export latest data to CSV
    csv_file = f"{category_name}_latest.csv"
    with open(csv_file, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["Name", "Price", "Link", "Image", "Rating", "ReviewCount", "Stock"])
        writer.writeheader()
        writer.writerows(new_snapshot)

    # Print alerts
    alerts = compare_snapshots(old_snapshot, new_snapshot)
    print(f"--- Alerts for {category_name} ---") if alerts else print(f"No changes detected for {category_name}")
    [print(alert) for alert in alerts]
```

When run periodically, this script will produce:

* A **CSV file** with the most recent product data per category.
* A **JSON snapshot** for historical comparison.
* **Console alerts** when products change stock or price.

Using this workflow, you can turn the one-off scrapers we built into a real monitoring system for Walmart data.

### Some Improvements You Can Do

The tracker we built stores snapshots in JSON and CSV, then prints alerts to the console.

That's enough for testing, but in production you'll want more robust workflows:

* **Database Merge:** instead of keeping separate JSON snapshots, feed each run into a central database (SQLite, PostgreSQL, MongoDB). Use a unique product identifier like the product link or SKU as the primary key. Each new run updates the record if it exists or inserts a new row, building a complete history of stock and price changes.
* **Email Alerts:** console logs are fine for debugging, but not practical for monitoring. Integrate an email service (SMTP, SendGrid, Amazon SES) so whenever the `compare_snapshots` function finds changes, you get an alert. Each email can include product name, old vs. new price, and stock transitions.
* **Scheduling:** running the tracker once is only a proof of concept. To make it useful, schedule it to run automatically. On Linux you can use **cron jobs**, on Windows use **Task Scheduler**, or in the cloud rely on AWS Lambda, Google Cloud Scheduler, or GitHub Actions.

## Or The Easy Way

Everything above parses HTML because that's what the `/store` endpoint returns: the raw Walmart page, fetched through a session already warmed for your store.

If you don't need the HTML, you don't have to parse it at all. The [Walmart Scraper API](/products/ready-api/walmart-scraper/) has two structured endpoints that do the parsing server-side and hand you JSON.

### Search Results

`/plugin/walmart/search` takes a keyword and a store, and returns the shelf as structured records:

```python
import requests

token = "<SDO-token>"

response = requests.get(
    "https://api.scrape.do/plugin/walmart/search",
    params={
        "token": token,
        "q": "milk",
        "store": "3520",
    },
)

data = response.json()
print(data)
```

**Sample response:**

```json
{
  "store": {
    "store_id": "3520",
    "city": "North Arlington",
    "state": "NJ",
    "postal_code": "07031"
  },
  "query": "milk",
  "total_results": 461,
  "results": [
    {
      "position": 4,
      "item_id": "10450114",
      "name": "Great Value Whole Vitamin D Milk, Gallon",
      "url": "https://www.walmart.com/ip/Great-Value-Whole-Vitamin-D-Milk-Gallon/10450114",
      "price": 4.23,
      "currency": "USD",
      "rating": 4.6,
      "reviews_count": 367220,
      "seller": "Walmart.com",
      "availability": "IN_STOCK",
      "fulfillment": ["pickup", "delivery"],
      "sponsored": false,
      "category": "Home Page/Food/Dairy & Eggs/Milk/Whole Milk"
    }
  ]
}
```

Sponsored placements are included and flagged with `"sponsored": true`, so you can keep or drop them yourself. Merchandising carousels that Walmart mixes into the same page ("trending", "highly rated") are excluded, because they don't answer the keyword you searched for.

### Product Details

`/plugin/walmart/product` takes the `item_id` from any search result and returns the full record for that item at that store:

```python
import requests

token = "<SDO-token>"

response = requests.get(
    "https://api.scrape.do/plugin/walmart/product",
    params={
        "token": token,
        "item": "10450114",
        "store": "3520",
    },
)

data = response.json()
print(data)
```

You get the price and stock for that store, plus the description, highlights, and the full specification table that only the product page carries:

```json
{
  "store": { "store_id": "3520", "city": "North Arlington", "state": "NJ" },
  "product": {
    "item_id": "10450114",
    "name": "Great Value Whole Vitamin D Milk, Gallon",
    "brand": "Great Value",
    "price": 4.23,
    "currency": "USD",
    "availability": "IN_STOCK",
    "fulfillment": ["delivery", "pickup"],
    "rating": 4.6,
    "reviews_count": 367220,
    "seller": "Walmart.com",
    "upc": "078742351865",
    "category": "Home Page/Food/Dairy & Eggs/Milk/Whole Milk"
  }
}
```

Because a search result's `item_id` is exactly what the product endpoint takes, the two chain directly: search a keyword, then look up whichever items you care about. Keep the same `store` across both calls and the price on the product page will match the price you saw in the search results.

### Why the Store Still Matters

Note the price above. That same gallon of Great Value milk is **$4.23** at store 3520 in New Jersey and **$2.92** at store 1735 in Illinois, on the same day.

This is the whole reason these endpoints require a store. Send a request without one and Walmart answers from whichever store it guessed, so you get a real price with no way to know which shelf it came from or how to reproduce it. Pass `store` when you're tracking fixed locations over time, or `zipcode` when you want to model what a shopper in a given area would see.

Both endpoints cost 10 credits per successful request, and failed requests are never charged. Full parameter and response reference is in the [Scrape.do Walmart Scraper Docs](/documentation/walmart-scraper-api/).

If you're weighing this against other providers, we benchmarked five Walmart APIs head to head in our [best Walmart scraper API](/blog/best-walmart-scraper-api/) comparison.

## Conclusion

Scraping Walmart in 2026 isn't simple.

Akamai firewalls, HUMAN CAPTCHAs, and store-specific cookies block basic scrapers fast.

But with the right setup that has valid cookies, referer-aware pagination, Scrape.do handling TLS and rendering, you can reliably extract:

* **Single products** (stock, price, discounts, details, images)
* **Variants** (sizes, packs, colors)
* **Categories** with pagination and duplicate filtering
* **Price and stock changes** over time

The same techniques apply whether you're focused on [scraping grocery delivery data](/blog/web-scraping-grocery-delivery-data/) from Walmart or expanding to other platforms for comprehensive [food delivery data scraping](/blog/web-scraping-food-delivery-data/).

👉 [Get started with Scrape.do, for FREE](https://dashboard.scrape.do/signup).