Headers & Cookies

Overview

HTTP headers and cookies are critical components for successful web scraping. Modern websites use headers to identify legitimate traffic and cookies to maintain session state. Scrape.do provides three distinct approaches to header management, each suited for different use cases.

Header Management Strategies

• Custom Headers — Gives you complete control over all headers sent to the target website. Use this when you need to replicate specific browser behavior or handle complex authentication flows.

• Extra Headers — Allows you to add specific headers on top of Scrape.do's automatically generated ones. This is ideal when you need to include one or two custom headers (like API keys or tracking IDs) without disrupting the optimized header set.

• Forward Headers — Strips away all auto-generated headers and forwards only what you explicitly provide. Perfect for scenarios where you're working with APIs or need minimal header overhead.

Cookie Management

• Set Cookies — Maintain session state across multiple requests. Use the setCookies parameter to pass authentication cookies, session tokens, or any other cookie-based data required by your target website.