Crawler

AWSpider is a web crawler for Amazon Web Services, written in Python #md5=edb1228e36df1a72e28bb07119e7a2e2

WebReaper is web crawler or spider, which can work its way through a website, downloading pages, pictures and objects that it finds so that they can be viewed locally, without needing to be connected to the Internet. The sites can be saved locally as a fully-browsable website which can be viewed...

Spiderman has been in many films, comics and adventures. By day Peter Parker, a young science student, Spiderman has millions of fans across the world, and he now actually forms a part of a new breed of superheroes called The New Avengers.If you think you are a fan of Spiderman, you can now...


Platforms: Windows

Nutch project is Web searching software which builds on Lucene Java, adding Web specifics such as a crawler, a link-graph database, parsers for HTML and other document formats, etc. Interesting files include: docs/api/index.html Javadocs for the Nutch software. CHANGES.txt Log of changes...


Platforms: *nix

KrawlSite is a web crawler/spider/ offline browser/download manager application. To integrate with Konqueror, open the file associations page in the configuration dialog, select text/html mime type and in the embedded viewers list choose KrawlSite_Part. Now when you right click on a web-page in...


Platforms: *nix

A stand-alone web crawler that generates xml sitemaps, finds broken links, and validates markup & css so you can maintain error free, fully indexed sites without sacrificing productivity. - Added free offers. - Made compatible with 32-bit Macs. - Modified scanner to ignore content received with...


Platforms: Mac

The web crawler is a program that automatically traverses the web by downloading the pages and following the links from page to page. A general purpose of web crawler is to download any web page that can be accessed through the links. This process is called web crawling or spidering. Many sites,...


Platforms: Windows, Mac, Linux, Linux Console, Linux Gnome, Linux GPL, Linux Open Source, Java

If you want to setup your computer system again you need to have the licenses and serial numbers at hand for all the software programs that you have purchased and registered. Instead of searching for the keys in your emails, manuals and recipes you could use another approach. License Crawler is a...


Platforms: Windows

Labyrinth of Zarus is a retro-themed adventure/dungeon crawler video game. You play as a treasure hunter searching for the lost treasures inside the labyrinth. Collect treasure, explore levels and fend off enemies with your whip. Each level is randomly generated offering a new experience each...


Platforms: Windows

Space Jammer is a crawler which shuffles through the artist-pages of MySpace and plays the music it finds there. You can create own playlists based on individual preferences for the music-search. If you want to get more information about an artist or if you want to buy his record, you can...


Platforms: Windows

Discontinued lightweight Desktop-Files/SMB/FTP crawler and search engine.


Platforms: Windows, Mac, Linux

JCrawler is a perfect cralwing/load-testing tool which is cookie-enabled and follows human crawling pattern (hit/second).


Platforms: Windows, Mac, Linux

MyNewsGroups :) is a Web based USENET news crawler, news reader and news poster. With the use of a DB backend, the crawler fetch the newsgroups messages ONCE only. Web based environment, SPAM Filters, Search Engine, Subscriptions and much more.


Platforms: Windows, Mac, Linux

An open source search engine and crawler based on best open source technologies: lucene, zkoss, tomcat, poi, tagsoup. A stable, high-performance piece of software. It is a modern search engine and a suite of high-powered full text search algorithms. OpenSearchServer will allow you to create any...


Platforms: Windows, Mac, Linux

PHPCrawl is a set of classes written in PHP for crawling/spidering websites, so just call it a webcrawler-library for PHP. The crawler "spiders" websites and delivers information about all found pages, links, files and so on to users of the library. By overriding a special method of the...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

SNT is a search engine for SMB and FTP shares with crawler running on Win32. Web interface is provided for searching files and browsing shares contents.Also provided shared films list with users rates and comments.


Platforms: Windows, Mac, Linux

This game is a mouse-driven dungeon crawler. The player moves the PC around, buys equipment, and attacks enemies using the mouse. More info on the wiki page. Use git to get the code.


Platforms: Windows, Mac, Linux

A very simple text based dungeon crawler, for Windows 32 bit (tested on Win XP). Fully playable though :) Written in python 2.7, as a way to learn the language. Everyone is free to use the code as they see fit.


Platforms: Windows, Mac, Linux

YACY is a p2p-based search engine for the WWW, based on it's distributed crawler and the built-in caching/scraping http proxy. Brings it's own database, has a httpd/web interface. Implemented in Java, supports all platforms.


Platforms: Windows, Mac, Linux

The application is based on other open source technologies like POI, Lucene, Zkoss and Tagsoup. Key features "OSS Open Search Server": - Multi-language indexing. Documents can be indexed in ten languages OCo English, French, German, Spanish, Italian, Portuguese, Finnish, etc. - Multi-lingual...


Platforms: Windows, Mac, *nix, Java, BSD Solaris