AI Published
What Is llms.txt? How AI Agents and Scrapers Use It
What llms.txt is, what goes in the file, how it differs from robots.txt and sitemap.xml, who serves one (we checked 55 sites), and how to read one with Python.
Author profile
Selman Gökçe
Head of Marketing
Nearly 10 years of experience in marketing. I currently lead marketing at SDO. Specialize in product marketing, content strategies, SEO & AEO.
I build AI pipelines and workflows to effectively create and scale marketing channels. New to web scraping, but that doesn't stop me from building production-ready content engines :)
Expertise map
What Selman has written, by topic. Pick a topic to jump to it.
Latest articles
The 6 articles Selman published or updated most recently.
Scraping Errors Published
Python Requests Retry: Backoff, Retry-After and Jitter Done Right
Retry failed Python requests the right way: urllib3 Retry with HTTPAdapter, Retry-After, exponential backoff with full jitter, tenacity, proxies and timeouts, all tested on a local server.
Scraping Errors Published
Error 418 I'm a Teapot: Why Servers Send It to Scrapers and How to Fix It
HTTP 418 I'm a teapot began as an April Fools' joke. Today Binance and some WAFs send it to bots. What it means for a scraper, and how to fix it.
Scraping Tools Published
Top 5 Decodo (Smartproxy) Alternatives for Proxies and Web Scraping in 2026
Five Decodo alternatives in two lanes: scraping APIs measured against Decodo's API on the same 18 domains, and residential proxies priced per GB and per 1,000 pages.
Scraping Use Cases Published
AnyWho.com Scraping: Extract People Search and Reverse Lookup Data
Scrape AnyWho.com with Python and Scrape.do: get past its AWS WAF user agent block, parse profiles with no JSON-LD, export name search results to CSV, and run free reverse phone lookups.
Anti Bot Published
What Is CAPTCHA? How It Works and 11 Types Explained
What a CAPTCHA is, how it scores you before it tests you, and the 11 CAPTCHA types you meet online, with real screenshots.
01 / 05
Scraping Tools
9 articles by Selman on Scraping Tools, newest first.
Top 5 Decodo (Smartproxy) Alternatives for Proxies and Web Scraping in 2026
Five Decodo alternatives in two lanes: scraping APIs measured against Decodo's API on the same 18 domains, and residential proxies priced per GB and per 1,000 pages.
Published
Best Bing SERP API in 2026: 5 Providers Benchmarked on 560 Live Searches
Microsoft killed the Bing Search API in August 2025. We ran 560 live searches across 9 providers to find what actually replaces it, measured on success rate, speed, price, and output format.
Published
Top 6 ScrapingAnt Alternatives for Web Scraping in 2026
ScrapingAnt's small proxy pool, thin free tier, and weak coverage on hard targets hold projects back. Here are 6 alternatives that scale better.
Published
Top 5 Apify Alternatives for Web Scraping and Crawling in 2026
Five Apify alternatives benchmarked on the same 20 domains: success rate, response time and cost per 1K against Apify's own numbers. Pick the right one.
Published
Best Google Trends APIs in 2026: We Load-Tested 7 Providers and Only 3 Answered
We load-tested 7 Google Trends API providers with thousands of live requests, only 3 answered. Compare measured latency, per-bundle pricing, and the schema traps.
Published
Best SERP APIs in 2026: 9 Google Search APIs Compared
Compare 9 Google Search APIs side by side: speed, cost, AI Overview detection, and output richness benchmarked across 25 queries. Find the best SERP API for your pipeline.
Updated
Best Google Shopping APIs in 2026: Only 3 of 16 Providers Actually Work
We tested every Google Shopping API live: which providers return real products in 2026, which endpoints are dead, and what filtering still works. With prices.
Updated
Best Google Maps APIs in 2026: We Benchmarked 5 Providers on 370+ Live Requests
We benchmarked 5 Google Maps scraping APIs on 370+ live requests. Compare success rates, speed, and cost per 1K to replace the pricey official Places API.
Updated
Show 1 more article on Scraping ToolsShow fewer
Best SerpApi Alternatives in 2026: 7 Cheaper SERP APIs Compared
Seven SerpApi alternatives benchmarked on speed, cost, AI Overview detection, and output richness. Scrape.do leads at 21.6x less per request with comparable SERP data.
Updated
02 / 05
Scraping Use Cases
7 articles by Selman on Scraping Use Cases, newest first.
AnyWho.com Scraping: Extract People Search and Reverse Lookup Data
Scrape AnyWho.com with Python and Scrape.do: get past its AWS WAF user agent block, parse profiles with no JSON-LD, export name search results to CSV, and run free reverse phone lookups.
Published
How to Scrape Google Trends with Python (both Explore & Trending Now pages)
Scrape Google Trends interest over time, related queries, regional data, and trending topics with Python. Extract structured data using internal API tokens and Scrape.do.
Updated
Scraping Google AI Overviews in 2026: Extract Summary Text, Source References, and Structured Data with Python
Scrape Google AI Overviews with Python using Playwright and Scrape.do SERP API. Extract summary text, source references, and structured data from AI-generated search results.
Updated
2 Ways to Scrape Gemini Responses and Sources (Free & Quick)
Scrape Google Gemini responses and grounding sources with a single Python request, no browser, account, or API key. Plus the Scrape.do Gemini endpoint for scale.
Published
Scrape Full Content from Google News with Python
Scrape Google News headlines, article metadata, and full content with Python. Extract titles, sources, dates, and article text using BeautifulSoup, SERP API, and Scrape.do.
Updated
How to Scrape Google Scholar: Extract Search Results and Author Profiles with Python
Scrape Google Scholar search results, author profiles, and citation data with Python. Extract paper titles, publication lists, and citation metrics using BeautifulSoup and Scrape.do.
Published
Scrape Google Search Results Using Python (Organic, Paid, FAQs, Related Terms)
Scrape Google Search results with Python using raw HTML parsing and structured SERP API. Extract organic links, search ads, FAQs, and related terms with Scrape.do.
Updated
What Is llms.txt? How AI Agents and Scrapers Use It
What llms.txt is, what goes in the file, how it differs from robots.txt and sitemap.xml, who serves one (we checked 55 sites), and how to read one with Python.
Published
Also filed under AI
04 / 05
Scraping Errors
2 articles by Selman on Scraping Errors, newest first.
Python Requests Retry: Backoff, Retry-After and Jitter Done Right
Retry failed Python requests the right way: urllib3 Retry with HTTPAdapter, Retry-After, exponential backoff with full jitter, tenacity, proxies and timeouts, all tested on a local server.
Published
Error 418 I'm a Teapot: Why Servers Send It to Scrapers and How to Fix It
HTTP 418 I'm a teapot began as an April Fools' joke. Today Binance and some WAFs send it to bots. What it means for a scraper, and how to fix it.
Published
What Is CAPTCHA? How It Works and 11 Types Explained
What a CAPTCHA is, how it scores you before it tests you, and the 11 CAPTCHA types you meet online, with real screenshots.
Published
