Orb Web Spider

ItSucks software is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates. The application also provides a swing GUI and a console interface. All backend functionalities are also available...

Fetchgals is a Web spider that locates and optionally downloads free pictures and movies from the internet. Today, commercial porn sites are advertised like this: somebody puts together a thumbnail gallery, i.e. a web page with small thumbnail pictures linking to free full-size images or...

Sphider is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is suitable for adding search functionality to small or medium sites (up to around 100,000 pages). It also works great as a tool for site analysis - finding broken links, gathering...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

PhpDig is a web spider and PHP search engine project, using a MySQL database and flat file support.PhpDig search engine can store textual content of indexed documents in files.PhpDig search engine builds a glossary with words found in indexed pages. On a search query, it displays a result page...


Platforms: PHP

A simple web spider with several recursion strategies. Home page is at http://github.com/slinkp/spydey. It doesn't do much except follow links and report status. I mostly use it for quick and dirty smoke testing and link checking. The only unusual feature is the ``--traversal=pattern`` option,...


Platforms: *nix

Spider Tracker is a WordPress plugin developed by Tony Fernandes of MnM-Designs.com with one purpose. That purpose is to track web spiders' activity on your WordPress blog. It features a friendly user interface allowing you to define which spiders to track and how many recent index crawls to...


Platforms: PHP

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own little breed of DMOZ and Google. ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited ammount of links and pages. Powered by...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

It is basically a program that can make you a search engine. It is a web crawler, has all the web site source code (in ASP, soon to be PHP as well), and a mysql database.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since this crawler seeks to...


Platforms: Windows, Mac, *nix, Java, BSD Solaris

Larbin is a Web crawler intended to fetch a large number of Web pages. It should be able to fetch more than 100 millions pages on a standard PC with much u/d. This set of PHP and Perl scripts, called webtools4larbin, can handle the output of Larbin.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

WWW::PkgFind watches Web sites, FTP sites, GIT repositories, etc. for new code releases, and downloads them. In other words, its like a Web spider tuned for downloading software packages and patches. WWW::PkgFind is also able to generate a queue of incoming packages, to allow subsequent...


Platforms: *nix

Visual SEO Studio is a desktop SEO Audit Tool, a Visual approach to make SEO tasks quick and easy. The program lets you Crawl one or more sites in parallel with its Interactive Web Spider, Spot on-site issues at a glance, Save Page Screenshot miniatures and see where the fold lays, visually...


Platforms: Windows, Windows 7, Windows Server

Pattern is a web mining module for the Python programming language. It bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine...


Platforms: *nix

SiteAnalyzer it is a web crawler to scan the websites and check their technical and of SEO-parameters for errors and correct them effectively. Key features of the SiteAnalyzer - Scanning of all pages of the site, as well as images, scripts and documents - Getting server response codes for...


Platforms: Windows, Windows 8, Windows 7, Windows Server

PHP FastFileCache caches output from dynamic PHP scripts, and stores them in files for fast retrieval under high server load.. WebSuck is a web spider for Java2 (C) 2001 Åke Wallebom


Platforms: *nix

PhpDig provides a very small PHP search engine that uses a MySQL database on backend. PhpDig is a very small PHP search engine that uses a MySQL database on backend. Features: indexes both static and dynamic pages; Spiders almost all links in html content, in hrefs, areamaps, frames; and Full...


Platforms: *nix

Lyntin project is a Python-based MUD client and development framework. Lyntin is an extensible Mud client and framework for the creation of autonomous agents, or bots. Lyntin is centered around Python, a dynamic, object-oriented, and fun programming language. Lyntin is written entirely in...


Platforms: *nix

In the time it takes to grab a glass of water, Pandora is able to hunt down files by spidering a URL or performing a keyword-based search. Its clear, intuitive interface guarantees you are underway in no time, and extensive preferences allow advanced users to flex their muscles. After a rapid...


Platforms: Mac

The WebPageLoader application was developed to be an internet download utility. It downloads web pages and entire sequences of files from the internet. A clipboard hook listens for any text copied to the clipboard, which resembles an URL or image filename. The application will present a dialog...


Platforms: Windows

Management-Ware Craigslist Emails Extractor software is a newCraigslist Classified Leads Finder. Internet has many sources of free leads that you can use to market your product or service. The Craigslist Contacts Finder is a powerful contact extraction tool that can provide you with professional...


Platforms: Windows, Edition 2005