Frequently Asked Questions

How can I reduce network traffic and improve performance?

You can reduce network traffic and improve your performance by using the accept-encoding: gzip, deflate, br header value in your requests. Scrape.do supports gzip, deflate, and brotli compression types.

What should I do when I get 502 status codes?

When you get 502 status codes, you should try again. If there is no problem specific to the target web page, our system expects you to achieve a success rate of over 85%+. If you fall below this success rate, please contact support and we will check your requests.

Should I use a queuing system for my scraping project?

Yes, it will always be better and easier for your business to have a queuing system in your web scraping project and manage failed requests in this queuing system.

How should I manage request timing to reduce errors?

Spreading your requests over a certain period of time can greatly reduce your error rates, and getting the maximum response you can get as soon as possible by using your simultaneous request limits at full capacity can also increase your success rates. For this, you can determine which one will be better for you by doing tests.

I need to forward headers to the target website. How can I do this?

You should read the Custom Headers and Extra Headers sections in our documentation.

How often does a proxy change?

Our system changes and rotates the proxy for every request made to the target website. If you need to use the same IP address across multiple requests, take a look at the Sticky Sessions feature.

Is 2MB response limit enough?

We respond to millions requests per day, and we can say with confidence that there has been no HTML content that has exceeded the 2M limit so far. Since the content from the target site is returned as compressed, it is practically impossible with HTML to have a compressed content that exceeds the 2MB limit for web pages.