List Headline Image
Updated by Jack Smith on Apr 26, 2017
jack-smith-6 jack-smith-6
10 items   1 followers   0 votes   54 views

Top 20 web crawler tools

In this post, I’d propose top 20 popular web crawlers around the web for your reference. You may find the most suited web crawler that’s tailored to your needs.

  • See more at: Octoparse Blog
Cyotek WebCopy Copy websites locally for offline browsing

Cyotek WebCopy is a free tool for copying full or partial websites locally onto your harddisk for offline viewing.

WebCopy will scan the specified website and download it's content onto your hardisk. Links to resources such as stylesheets, images, and other pages in the website will automatically be remapped to match the local path. Using its extensive configuration you can define which parts of a website will be copied and how.

Octoparse| Free Web Scraping Tool

·Free Web Scraping Tool & Free Web Crawlers for Data Extraction without coding ·Cloud-Based Web Crawling ·Data As A Service

Version 3.49-1 (04/01/2017)

HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility. It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the 'mirrored' website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. HTTrack is fully configurable, and has an integrated help system. WinHTTrack is the Windows 2000/XP/Vista/Seven/8 release of HTTrack, and WebHTTrack the Linux/Unix/BSD release.


Download Getleft for free. Getleft is a Web site grabber, it downloads complete web sites according to the options set by the user.


Scraper gets data out of web pages and into spreadsheets.

OutWit Hub

To try the thousands of add-ons available here, download Mozilla Firefox, a fast, free way to surf the Web!

Everything you need for web scraping

Trying to get data from a complex and laggy sites?
No worries! Collect and store data from any JavaScript and AJAX page.

Scrapinghub: Web Crawling Platform & Data as a Service

Scrapy Cloud, our cloud-based web crawling platform, allows you to easily
deploy crawlers and scale them on demand – without needing to worry about
servers, monitoring, backups, or cron jobs. It helps developers like you
turn over two billion web pages per month into valuable data. - web data extraction tool for professionals

Extract, Enrich & Connect ANY data. Web scraping, data extraction and big data refinery tool for professionals