Web Crawler

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own little breed of DMOZ and Google. ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited ammount of links and pages. Powered by...

The Web Curator Tool (WCT) is an open-source workflow management application for selective web archiving. It is designed for use in libraries and other collecting organisations, and supports collection by non-technical users while still allowing complete control of the web harvesting process. It...

Fresh WebSuction is designed for downloading Web pages or web contents (reference material, software files, online books, e-zines, news articles) for viewing offline, and saving money on Internet connection fees. If you pay fees for the amount of time you are online you can save money by quickly...


Platforms: Windows

Octoparse is a powerful but easy-to-use and free web scraping tool that is being used by tens of thousands of individuals and companies all over the world. The word of Octoparse is the combination of the two words, “Octopus” and “Parse”....


Platforms: Windows, Windows 7

PHPCrawl is a set of classes written in PHP for crawling/spidering websites, so just call it a webcrawler-library for PHP. The crawler "spiders" websites and delivers information about all found pages, links, files and so on to users of the library. By overriding a special method of the...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Wayback Machine is an open source java implementation of the The Internet Archive Wayback Machine. The current production version of the Wayback Machine is implemented in perl, and lacks in maintainability and extensibility. Also, the code is not open source. Primary motivation for the new...


Platforms: *nix

External Site Catalog allows you to index and search external sites in a Plone site. ExternalSiteCatalog is a web crawler that can index external sites and make them searchable in Plone. You can specify the sites to index in a Plone Configlet, and directly index them from Plone, or let a...


Platforms: *nix

Desktop IRIS is an easy-to-use search program that can be successfully downloaded and accessed by anyone. It allows you to intuitively find stored information from your desktop and network without imposing any restrictions on the number of files and folder locations indexed. The system uses the...


Platforms: Windows, Mac

Netifera is a new modular open source platform for creating network security tools. This project provides many advantages for both security developers and researchers who want to implement new tools as well as the community of users of these tools. * TCP and UDP network scanning and service...


Platforms: Mac, Linux

This plugin was born because there was a no other real alternative to enable content sparationd and some CMS like functionalities in WordPress. The main goal was, to enhance WordPressd-deOaos functionalities, to hide some unwanted categories, from defined parts of the blog.Today, ACE can override...


Platforms: PHP

This module is under active development for a Drupal 6 upgrade have a look at the roadmap/progress Nutch RoadmapNutch is a web crawler/indexer/search engine that is based on Lucene. It is a Java tool.This module allows you to have basic control over the Nutch crawl lifecycle through the Drupal...


Platforms: PHP

Xapian is an Open Source Probabilistic Information Retrieval library, released under the GPL. Xapian iss written in C , with bindings to allow use from other languages (Perl, Java, Python, PHP, and TCL are currently supported; Guile and C# are being worked on). Xapian is designed to be a highly...


Platforms: *nix

Majento SiteAnalyzer is a web crawler (web spider) to scan the websites and check their technical and of SEO-parameters for errors and correct them effectively. Key features - Scanning of all pages of the site, as well as images, scripts and documents - Obtaining server response codes for...


Platforms: Windows, Windows 8, Windows 7, Windows Server

Schema Crawler project is a platform (database and OS) independent command-line tool to output your database schema and data in a readable form. The output is designed to be diff-ed with previous versions of your database schema. Schema Crawler is also an API that improves on standard JDBC...


Platforms: *nix

Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.


Platforms: Windows

Audit your website security with Acunetix Web Vulnerability Scanner As many as 70% of web sites have vulnerabilities that could lead to the theft of sensitive corporate data such as credit card information and customer lists. Hackers are concentrating their efforts on web-based applications -...


Platforms: Windows

Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net


Platforms: Windows, Mac, Solaris, Linux

Web Accounts Plus is a great utility that allows users to maintain their ever growing list of web accounts, credit cards, and frequent traveler information for free ! The information can be password protected and the application can be hidden in the system tray. You can drag and drop...


Platforms: Windows

A free WYSIWYG editor for the design of Web content using HTML and SVG (Scalable Vector Graphics). Features include an advanced Text Editor with spell checker and link editor, vector graphics editor, a Renderer with support for radial shading, linear shading and transparency, enhanced page layout...

Web Roller records a real user activity in terms of HTTP requests and creates a simple script. You can modify the script using embedded test data generating capabilities. You can set various parameters for requests and the script, and then run script or a batch of scripts. It creates a...


Platforms: Windows