Configurable range of concurrent requests to permit buyers to crawl their Site with no incurring supplemental processing. This may be configured from 1 to 16 concurrent requests. Assistance for Robots.txt, allowing for you to definitely customize the destinations the place the crawler ought to review and which locations ought to be dismissed. Assis