How to Scrape Amazon Products, All Variants, and Search Results with Python
Amazon is the world’s largest marketplace and go-to platform for every e-commerce vendor, especially in the US.
That’s why scraping Amazon can provide invaluable data and it’s the same reason why scraping it is so difficult…
… but not impossible.
In this guide, we’ll scrape simple product pages, product pages with multiple options and numerous variants, and search results + categories from Amazon; unlocking access to millions of Amazon products.
[Plug-and-play codes on our GitHub repo 🧰]
Looking to scrape Amazon reviews?
Does Amazon Allow Scraping?
Yes, scraping Amazon is possible and legal as long as you follow the guidelines and stay away from any login protected personal information.
By respecting these rules and ethical scraping practices, you can gather publicly available data without crossing any laws or regulations.
Legal Considerations
Web scraping is a widely used and legitimate method for collecting publicly available data.
However, every website, including Amazon, establishes boundaries to protect its resources. These are outlined in Amazon’s robots.txt file.
This file specifies which parts of the website are accessible to automated tools like web crawlers.
When scraping Amazon, it’s important to:
- Avoid login-protected content like extended customer reviews. These areas are restricted by Amazon’s Terms of Service and often require authentication.
- Respect the structure and limitations outlined in
robots.txtto ensure compliance.
⚠ In recent updates, Amazon moved extended customer reviews behind login walls, making them inaccessible without proper authorization. While these changes may pose challenges, they also clarify which areas of Amazon’s site are off-limits.
Ethical Guidelines
Scraping ethically means balancing your goals with respect for the website’s integrity and policies.
Ethical scraping doesn’t just help you avoid potential legal pitfalls; it also fosters a responsible approach to data collection.
Here are a few best practices:
- Respect Rate Limits: Sending too many requests in a short period can overwhelm Amazon’s servers. Use delays or throttling to mimic real user behavior.
- Stick to Publicly Available Data: Avoid scraping content that requires login credentials or is marked as off-limits in the
robots.txtfile. - Use Data Responsibly: Whether for analysis or application development, ensure that the data you collect is used in ways that align with Amazon’s guidelines.
To learn more about ethical scraping and the legal landscape, check out this detailed guide on legality of web scraping.
Bypass Amazon AWS WAF
Amazon uses its in-house solution, AWS WAF to protect the platform from bots and scrapers.
Any request you send, whether it’s through your Chrome browser or using Python Requests, your IP, headers, and recent activity is analyzed to make sure you’re human.
So, how do you make your bot bypass it?
Here are your options, basically:
- Manual methods/in-house solutions: create and maintain a pool of premium proxies and headers to make sure your requests are not flagged.
- Open-source stealth plugins: pick from countless options such as
scrapy-fake-useragent+scrapy-rotating-proxies,camoufox+cheap rotating proxies,rebrowser-puppeteerto go through AWS WAF. These would run locally and cause performance and success rate issues as you scaled, though. - (Method we will also use in this guide) Web scraping APIs: a web scraping API known to handle Amazon persistently such as Scrape.do will handle proxies, headers, and CAPTCHAs for you.
Amazon is not impossible to break for a few scraping requests.
But if you’re planning on extracting data from millions of products, you need to think about scalability, as re-using proxies and headers will only get you so far.
Scraping Product Data from Amazon (Basic)
Extracting information from an Amazon product page will be our starting point for this tutorial.
From product prices and images to descriptions and ratings, these pages contain data that can be invaluable for market analysis or application development.
For the scope of this step we will be using this product as an example.
Prerequisites
Libraries like Requests and BeautifulSoup simplify the process of sending HTTP requests and parsing HTML content.
pip install requests beautifulsoup4
Now that we have the necessary libraries installed we can start making our first request.
As we have talked about it earlier we will be using Scrape.do for bypassing the Amazon WAF and the last thing we will need is our token provided by Scrape.do
As we start, we will focus on getting a successful HTTP response from the Amazon servers. We will route our HTTP request through Scrape.do API which will allow us to effortlessly skip Amazon WAF.
This step normally would need to be quite complex due to advanced protection systems that are in use by Amazon but thanks to Scrape.do we don’t need to handle any bypass mechanisms!
After defining our token and the product url, we can try sending our first request using something like this:
import requests
# Our token provided by 'Scrape.do'
token = "<your_token>"
# Amazon product url
targetUrl = "https://us.amazon.com/Amazon-Basics-Portable-Adjustable-Notebook/dp/B0BLRJ4R8F/"
# Use scrape.do to get contents using your token
apiUrl = "http://api.scrape.do?token={}&url={}".format(token, targetUrl)
response = requests.request("GET", apiUrl)
print(response)
At this point we should be getting a HTTP response stored in our response variable and if we print it into our console it should be looking like this:
<Response [200]>
HTTP response with the 200 code means our request is successful.
We can access the contents of the response by using;
response.text
This will include all of the html contents of the page including any scripts that is available.
If you want; you can print this information on your console to check or even write it into a html file to have a local copy of the product page on your computer.
Our next step will be extracting the information we need from this raw text.
How to Scrape Amazon Product Prices
Price is the most important information about a product, so it will be a nice starting target for us.
For this step we will start using BeautifulSoup because it helps us parse the data we need.
Before starting the extraction process we need to locate the information that is necessary inside the product page we just downloaded. We can either use the response.text that we downloaded in previous step or we can go into the product page and use developer tools to inspect the elements.
You can use F12 to access the developer tools on Google Chrome and press ctrl + shift + c to inspect the element you want.
Or you can click the inspection tool:
Hover over the price details and you will see that the price is divided into whole section and fraction section:
We will use BeautifulSoup to extract price details from both these sections and then put them together:
⚠ Amazon uses different price structures depending on product availability. We need to handle both regular pricing and out-of-stock scenarios, so we’ll first see if the item is out of stock and scrape the price only if it isn’t.
from bs4 import BeautifulSoup
<-- same as previous step -->
# Parse the request using BS
soup = BeautifulSoup(response.text, "html.parser")
# Extract price with out-of-stock handling
if soup.find("div", id="outOfStockBuyBox_feature_div"):
price = "Out of Stock"
else:
whole = soup.find(class_="a-price-whole")
fraction = soup.find(class_="a-price-fraction")
price = f"${whole.text}{fraction.text}"
print("Price:", price)
After running this script we should be getting the product price printed on our console, whether it’s available or out of stock.
Price: $28.57 or Price: Out of Stock
Scrape Other Product Details
The following steps will be pretty similar to getting the price information.
We will inspect each of the information that we are insterested using the developer tools and we will try to find an encapsulating html object that holds the complete information.
Then with the help of BeautifulSoup we will locate this information inside the response we are getting.
Product Name
We can see that product name is stored inside a span object with the id productTitle. Again we will access this object with BeautifulSoup and try to print the information inside of it.
name = soup.find(id="productTitle").text.strip()
print("Product Name:", name)
We should be getting product name printed on our console.
Product Name: Amazon Basics Ergonomic and Foldable Laptop Stand for Desk, Adjustable Riser, Fits all Laptops and Notebooks up to 17.3 Inch, 10 x 8.7 x 6 in, Silver
Product Image
It is time for the product image; we will start with inspecting the element again and find the html object containing the product image.
landingImage is the id of the element that contains the image.
But this time we are interested in the src attribute of the object instead of the text, since image url is stored in this attribute. We can find and access to the src attribute of this element using following:
image = soup.find("img", {"id": "landingImage"})["src"]
print("Image URL:", image)
We should be getting the image url printed on our console!
Image URL: https://m.media-amazon.com/images/I/51KyaTB1EKL.__AC_SX300_SY300_QL70_FMwebp_.jpg
It is also possible to download and save this images into your drive but for the scope of this tutorial we will stop here and store the information about product images as urls.
Product ASIN
The ASIN (Amazon Standard Identification Number) is a unique identifier for each product. We can extract this directly from the URL, which is useful for database storage and referencing.
# Extract ASIN from URL
asin = targetUrl.split("/dp/")[1].split("/")[0].split("?")[0]
print("ASIN:", asin)
This will give us:
ASIN: B0BLRJ4R8F
Product Rating
And finally the product rating.
Our first step is again to inspect:
As we can see product rating element is not encapsulated directly under a tag that we can refer to. a-size-medium a-color-base classes are generic classes and will return other objects if we tried to use it.
div object with the AverageCustomerReviews class is something we can easily access and contains all of the information we need but also has information that we don’t need.
We will try to get this information and strip the part we are interested.
rating = soup.find(class_="AverageCustomerReviews").text.strip()
print("Rating:", rating)
This will return the following:
Rating: 4.6 out of 5 stars4.6 out of 5
This string contains the rating twice, also we don’t need to specify it is out 5.
So we are going to split this string and get a single rating score. Let’s update our code as follows:
rating = soup.find(class_="AverageCustomerReviews").text.strip().split(" out of")[0]
print("Rating:", rating)
Now we get this:
Rating: 4.6
Export to CSV
To make your scraped data actionable and be able to easily scale this up to a thousand products, it’s essential to save it in a structured data format.
So we will export it into CSV which is the most basic format.
You can use libraries like pandas to easily save your data frames into csv files but in this tutorial we will do it without a help of a library, using Python’s built-in module.
Let’s also use two additional products to make our code loop through URLs.
Here’s the final code:
import requests
from bs4 import BeautifulSoup
# Our token provided by 'Scrape.do'
token = "<your-token>"
# Amazon product urls
targetUrls = ["https://us.amazon.com/Amazon-Basics-Portable-Adjustable-Notebook/dp/B0BLRJ4R8F/",
"https://us.amazon.com/Urmust-Ergonomic-Adjustable-Ultrabook-Compatible/dp/B081YXWDTQ/",
"https://us.amazon.com/Ergonomic-Compatible-Notebook-Soundance-LS1/dp/B07D74DT3B/"]
for targetUrl in targetUrls:
# Use scrape.do to get contents using your token
apiUrl = "http://api.scrape.do?token={}&url={}".format(token, targetUrl)
response = requests.request("GET", apiUrl)
# Parse the request using BS
soup = BeautifulSoup(response.text, "html.parser")
name = soup.find(id="productTitle").text.strip()
# Extract ASIN from URL
asin = targetUrl.split("/dp/")[1].split("/")[0].split("?")[0]
# Extract price
if soup.find("div", id="outOfStockBuyBox_feature_div"):
price = "Out of Stock"
else:
whole = soup.find(class_="a-price-whole")
fraction = soup.find(class_="a-price-fraction")
price = f"${whole.text}{fraction.text}"
image = soup.find("img", {"id": "landingImage"})["src"]
rating = soup.find(class_="AverageCustomerReviews").text.strip().split(" out of")[0]
with open("output.csv", "a") as f:
f.write('"' + asin + '", "' + name + '", "' + price + '", "' + image + '","' + rating + '" \n')
And voila, information about these 3 products should be saved into our CSV file.
Here’s our output:
Scrape All Color,Size,Model Variations of an Amazon Product (Advanced)
Many Amazon products don’t have a single, static price.
The price changes with the option you choose: color, size, style, model and more.
If you only scrape the default page, you’ll miss most of the catalog, so the right way is to discover every available variation and visit each variation’s dedicated URL to read its own price and selected options.
What we’re building below is a reliable (and maybe overengineered) system that:
- Finds all variation dimensions on the base product page (color, size, style, etc.)
- Walks through every valid option combination without duplication
- Reads the selected options on each visited page and grabs prices and product names
- Writes everything into a clean CSV
I’m probably picking the most complicated target page possible for this code, but we need our code to work in complex scenarios, so it’ll be good practice :)
We’ll scrape all colors, storage sizes, and service providers for Apple iPhone 14 Plus here:
Extract All Available Variations
We’ll begin by grabbing the base product page and mapping out the full set of variation dimensions and their options.
First, we build the API request through Scrape.do:
import requests
import urllib.parse
import csv
from bs4 import BeautifulSoup
import re
# Scrape.do token and target URL
token = "<your-token>"
PRODUCT_URL = "https://us.amazon.com/Apple-iPhone-Plus-128GB-Blue/dp/B0CG84XR6N/"
# Initial page scrape to find dimensions
api_url = f"http://api.scrape.do?token={token}&url={urllib.parse.quote_plus(PRODUCT_URL)}&geoCode=us"
soup = BeautifulSoup(requests.get(api_url).text, "html.parser")
Scrape.do returns the fully-rendered HTML and handles the heavy lifting (WAFs, geo, headers).
Then, we’ll reuse the exact same product name and price logic from previous section: if the out-of-stock container exists, we mark it as such; otherwise we stitch the whole and fractional price.
name = soup.find(id="productTitle").text.strip()
price = "Out of Stock" if soup.find("div", id="outOfStockBuyBox_feature_div") else f"${soup.find(class_='a-price-whole').text}{soup.find(class_='a-price-fraction').text}"
Then, we’ll extract all dimension and option names from the select menus.
# Extract available dimensions and options
dimensions = {}
for row in soup.find_all("div", {"id": re.compile(r"inline-twister-row-.*")}):
dim_name = row.get("id", "").replace("inline-twister-row-", "").replace("_name", "")
options = []
for option in row.find_all("li", {"data-asin": True}):
asin = option.get("data-asin")
if asin and option.get("data-initiallyUnavailable") != "true":
swatch = option.find("span", {"class": "swatch-title-text-display"})
img = option.find("img")
button = option.find("span", {"class": "a-button-text"})
option_name = swatch.text.strip() if swatch else img.get("alt", "").strip() if img and img.get("alt") else button.get_text(strip=True) if button and button.get_text(strip=True) != "Select" else "Unknown"
options.append({"name": option_name, "asin": asin})
if options:
dimensions[dim_name] = options
print(f"Found dimensions: {list(dimensions.keys())}")
for dim_name, options in dimensions.items():
print(f" {dim_name}: {len(options)} options")
Amazon exposes variation groups in “inline-twister-row-*” containers. Each option usually has:
- A display label from a swatch title, image alt, or button text
- An ASIN we can navigate to (the key to get the correct price for that option)
We skip options that are initially unavailable to avoid dead ends.
We then also set up a sensible traversal order and CSV headers so downstream steps are clean and deterministic.
# Setup dimension traversal order and CSV headers
priority = ['color', 'size', 'style', 'pattern', 'material', 'fit']
dim_names = sorted(dimensions.keys(), key=lambda x: priority.index(x.lower()) if x.lower() in priority else len(priority))
headers = ["ASIN", "Product Name"] + dim_names + ["Price"]
results = []
scraped = set()
Scrape All Product Variations One by One
Next, we’ll recursively walk through the option tree.
For each option, we load its dedicated product page (using the option’s ASIN) and capture both the selected options and the correct price.
# Recursive function to scrape all product variations
def scrape_variations(asin, dim_index=0, prefix=""):
if asin in scraped:
return
url = f"{PRODUCT_URL.split('/dp/')[0]}/dp/{asin}/?th=1&psc=1"
api_url = f"http://api.scrape.do?token={token}&url={urllib.parse.quote_plus(url)}&geoCode=us"
soup = BeautifulSoup(requests.get(api_url).text, "html.parser")
name = soup.find(id="productTitle").text.strip()
price = "Out of Stock" if soup.find("div", id="outOfStockBuyBox_feature_div") else f"${soup.find(class_='a-price-whole').text}{soup.find(class_='a-price-fraction').text}"
page_dims = {}
for row in soup.find_all("div", {"id": re.compile(r"inline-twister-row-.*")}):
dim_name = row.get("id", "").replace("inline-twister-row-", "").replace("_name", "")
options = []
for option in row.find_all("li", {"data-asin": True}):
opt_asin = option.get("data-asin")
if opt_asin and option.get("data-initiallyUnavailable") != "true":
swatch = option.find("span", {"class": "swatch-title-text-display"})
img = option.find("img")
button = option.find("span", {"class": "a-button-text"})
option_name = swatch.text.strip() if swatch else img.get("alt", "").strip() if img and img.get("alt") else button.get_text(strip=True) if button and button.get_text(strip=True) != "Select" else "Unknown"
options.append({"name": option_name, "asin": opt_asin})
if options:
page_dims[dim_name] = options
Every time we navigate to an option’s ASIN, the page’s available options can change (e.g., some sizes are only available for certain colors).
We always read the page’s current dimensions before deciding what to do next, and then record selected options and price from the current page:
# End of recursion - collect final data
if dim_index >= len(dim_names) or not page_dims:
scraped.add(asin)
selections = {}
for dim_name in dim_names:
for row_id in [f"inline-twister-row-{dim_name}_name", f"inline-twister-row-{dim_name}"]:
row = soup.find("div", {"id": row_id})
if row:
selected = row.find("span", {"class": re.compile(r".*a-button-selected.*")})
if selected:
swatch = selected.find("span", {"class": "swatch-title-text-display"})
img = selected.find("img")
selections[dim_name] = swatch.text.strip() if swatch else img.get("alt", "").strip() if img and img.get("alt") else selected.get_text(strip=True) if "Select" not in selected.get_text(strip=True) else "N/A"
break
else:
for option in row.find_all("li", {"data-asin": asin}):
swatch = option.find("span", {"class": "swatch-title-text-display"})
button = option.find("span", {"class": "a-button-text"})
selections[dim_name] = swatch.text.strip() if swatch else button.get_text(strip=True) if button and "Select" not in button.get_text(strip=True) else "N/A"
break
break
row = [asin, name] + [selections.get(dim, "N/A") for dim in dim_names] + [price]
results.append(row)
sel_str = ", ".join([f"{k}:{v}" for k, v in selections.items()])
print(f"{prefix}✓ {asin}: {price} | {sel_str}")
return
And at a terminal node, we compile a single output row that includes:
- The ASIN we visited (unique identifier for the variation)
- The product title
- The selected option values for each dimension
- The price shown on that page
And then, we repeat it. Move along to the next dimension and visit each option once:
# Continue recursion through dimensions
current_dim = dim_names[dim_index]
if current_dim in page_dims:
options = page_dims[current_dim]
if dim_index == 0:
print(f"{prefix}Found {len(options)} {current_dim} options")
for i, option in enumerate(options):
if dim_index == 0:
print(f"{prefix}{current_dim} {i+1}/{len(options)}: {option['name']}")
scrape_variations(option["asin"], dim_index + 1, prefix + " ")
else:
scrape_variations(asin, dim_index + 1, prefix)
We use a simple depth-first traversal.
The scraped set ensures we never visit the same ASIN twice.
For the first dimension, we print progress so we’ll be able to track progress too.
Run Code and Export
We start the crawl from the ASIN present in the base URL and write rows as we complete leaves.
Headers are constructed dynamically from discovered dimensions so your CSV is always aligned with each product’s real variation model:
# Start scraping process
original_asin = PRODUCT_URL.split("/dp/")[1].split("/")[0].split("?")[0]
print(f"\nDimension traversal order: {dim_names}")
print(f"\nStarting variation crawling...")
scrape_variations(original_asin)
And finally, we’re exporting all results to a CSV:
# Save results to CSV
output_file = "amazon_variations.csv"
with open(output_file, 'w', newline='', encoding='utf-8') as f:
writer = csv.writer(f)
writer.writerow(headers)
writer.writerows(results)
When you run the complete code, it’ll start printing progress on the terminal:
Found dimensions: ['service_provider', 'color', 'size']
service_provider: 7 options
color: 7 options
size: 4 options
Dimension traversal order: ['color', 'size', 'service_provider']
Starting variation crawling...
Found 7 color options
color 1/7: (PRODUCT)Red
✓ B0BN969PGL: $389.99 | color:(PRODUCT)Red, size:128GB, service_provider:AT&T
✓ B0DK2KTXJT: $399.00 | color:(PRODUCT)Red, size:128GB, service_provider:Boost Mobile
✓ B0BN95F27D: $389.99 | color:(PRODUCT)Red, size:128GB, service_provider:GSM Carriers
...
<--- omitted --->
...
Writing 89 results to amazon_variations.csv
Done! Results saved to amazon_variations.csv
Scraped 89 unique variations
It’s easier said than done, but that’s it! And here’s what your CSV file will look like:
A clean, machine-readable CSV with one row per unique variation, each row tied to the actual ASIN, the exact selected options, and the price shown for that combination.
Scrape Amazon Categories and Search Results
Amazon’s categories for products and search results function exactly the same, so we’ll combine the tutorial for the two processes under one tutorial.
We’ll scrape product information and navigate through multiple pages of search results; parsing individual product data and handling pagination.
For this example we will use the laptop stands category.
We can either use the next-page button to iterate through the results or we can request each result page by adding this information in to the URL we are requesting.
Using the next-page button, while sounding more reasonable, might get blocked by Amazon because they are checking these navigation options for bot traffic. Plus, it will consume more resources as it will require us to render the page.
Just to make sure we are not blocked while trying to navigate between pages we will request each search page one-by-one until we reach maximum amount of pages
Let’s start with our definitions:
import urllib.parse
import csv
import requests
from bs4 import BeautifulSoup
# Our token provided by 'scrape.do'
token = "<your_token>"
current_result_page = 1
max_result_page = 20
# Initialize list to store product data
all_products = []
Since we have 20 result pages for this category we set our max_result_page variable as 20.
Scrape Result Pages
We have everything we will need ready and we will loop through the different category pages now. We will break this loop once we reach the maximum amount of pages.
Each result element has the s-result-item as their class. We will go through all of theses results and scrape their name, price, link and images.
You can see the necessary selectors for each attribute defined below, and our code for this part should look like this:
<----- same as previous step ------>
# Loop through all result pages
while True:
# break the loop when max page number is reached
if current_result_page > max_result_page:
break
targetUrl = urllib.parse.quote("https://www.amazon.com/s?k=laptop+stands&page={}".format(current_result_page))
apiUrl = "https://api.scrape.do/?token={}&url={}".format(token, targetUrl)
response = requests.request("GET", apiUrl)
soup = BeautifulSoup(response.text, "html.parser")
# Parse products on the current page
product_elements = soup.find_all("div", {"class": "s-result-item"})
for product in product_elements:
try:
# Extract product details
name = product.select_one("h2 span").text
try:
price = str(product.select("span.a-price")).split('a-offscreen">')[1].split('</span>')[0]
except:
price = "Price not available"
link = product.select_one(".a-link-normal").get("href")
image = product.select_one("img").get("src")
# Append data to the list
if name:
all_products.append({"Name": name, "Price": price, "Link": link, "Image": image})
except:
continue
current_result_page += 1
And, here it is, we stored all of the product information into all_products variable. Only thing we need to do now is export them into a file.
Export Data to CSV
We will use CSV library to save all of this information into a file, we can achieve that as follows:
# Export the data to a CSV file
<----- same as previous step ------>
csv_file = "amazon_search_results.csv"
headers = ["Name", "Price", "Link", "Image"]
with open(csv_file, "w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(all_products)
In the end our amazon_search_results.csv file should be looking like this:
Conclusion
Amazon has millions of products competing with each other, and if you’re trying to compete in the e-commerce stage you need the right data to generate actionable insights or set up automations.
And at the end of the day, whether you’re scraping just Amazon or multiple e-commerce sites, getting blocked is the last thing you’ll want.
This is where Scrape.do comes in the picture.
While you focus on taking impactful actions on the data you’ve scraped, Scrape.do handles:
- Automated proxy rotation with 100M+ datacenter, mobile, and residential IPs,
- Avoiding or solving CAPTCHAs
- Handling TLS fingerprinting, header rotation, and user agents,
- Monitoring and validating responses.
All at a fraction of the cost of doing all these yourself.
Start scraping today with 1000 free credits.
Frequently Asked Questions
Can you scrape Amazon for prices?
Answering this question from two different angles; yes, it is legal to scrape product prices on Amazon because they’re public data and yes, even a beginner developer can easily scrape them following this guide using Python.
How do you retrieve data from Amazon?
You can easily retrieve data from Amazon in 3 steps:
- Use a web scraping API or a stealth plugin to bypass Amazon’s firewall,
- Write a code in your preferred programming language that retrieves HTML responses and parses the data you want (more info in this guide)
- Input your target URLs and automate the process using workflows.
Is Amazon easy to scrape?
Amazon is not easy to scrape at all. If you’re trying to scrape without an existing scraping operation to handle anti-bot restrictions and proxies you’ll instantly get blocked. If you use a scraping API that is known to handle Amazon’s WAF, however, it can become quite simple to extract data from Amazon.
Does Amazon have anti-scraping?
Yes, Amazon uses AWS WAF as it’s firewall to block all bots including scrapers. It analyzes your IP, headers, and activity to make sure you’re a regular human browsing the website; and if you’re not you’re blocked instantly.
Software Engineer
Full-stack software developer with 10+ years of experience. Relatively fresh to web scraping, and sharing my experience as I go on the Scrape.do blog.