When is it necessary to adjust extraction speed?

There are websites that receive a very large number of visitors simultaneously around the world. Simply put, websites like YouTube and Amazon have a very high number of concurrent users.

When collecting data from such sites, if the collection speed is set too fast, the website may consider the visits abusive and restrict data extraction. In other words, data extraction can fail. The user's IP address may also be blocked.

At Restly, to prevent extraction failures, we provide a blue bar at the top right of the dashboard so users can adjust the data extraction execution speed.

1. Reduce extraction speed

---

Data collection speed refers to the number of URLs collected simultaneously. In other words, "extraction speed = 15" can be interpreted as collecting up to 15 web pages at the same time. From the perspective of the website providing the data, this can be a traffic attack. If too many visits occur to a website in a short time, many sites will try to determine whether the visitor is a bot and may present a security captcha. They may also block the IP address and permanently suspend access to the site.

To prevent and mitigate such issues, Restly provides a feature that allows users to adjust the extraction speed themselves. The recommended default extraction speed is 1 or 2. In that case, the pace is similar to a person visiting the website and collecting data manually. As the collection speed slows, the likelihood of extraction failure decreases.

1. Go to the dashboard and click the execution speed bar at the top right.

2. Adjust to the desired extraction speed, then press the [Yes] button. If you want to ensure data is collected even if the extraction speed is slow, set it to the slowest speed, 1.

Selecting an extraction speed of 1 collects at the slowest rate, while selecting 15 collects at the fastest rate. The difference is whether you collect 1 URL at a time or 15 URLs at a time.

2. Increase extraction speed

---

If you want to complete data collection quickly, you can increase the extraction speed at your discretion. If the extraction speed is 7, it means up to 7 web pages are collected simultaneously.

However, as noted earlier, the faster the extraction speed, the higher the chance of your IP address being blocked. There is a way to resolve this even if your IP is blocked. Restly offers an option to pay an additional fee to use a dedicated purchase a private proxy serverand that will solve the issue.