Crawler

PHPCrawl is a set of classes written in PHP for crawling/spidering websites, so just call it a webcrawler-library for PHP. The crawler "spiders" websites and delivers information about all found pages, links, files and so on to users of the library. By overriding a special method of the...

SNT is a search engine for SMB and FTP shares with crawler running on Win32. Web interface is provided for searching files and browsing shares contents.Also provided shared films list with users rates and comments.

This game is a mouse-driven dungeon crawler. The player moves the PC around, buys equipment, and attacks enemies using the mouse. More info on the wiki page. Use git to get the code.


Platforms: Windows, Mac, Linux

A very simple text based dungeon crawler, for Windows 32 bit (tested on Win XP). Fully playable though :) Written in python 2.7, as a way to learn the language. Everyone is free to use the code as they see fit.


Platforms: Windows, Mac, Linux

YACY is a p2p-based search engine for the WWW, based on it's distributed crawler and the built-in caching/scraping http proxy. Brings it's own database, has a httpd/web interface. Implemented in Java, supports all platforms.


Platforms: Windows, Mac, Linux

Welcome to Sword of Fargoal® Legends, **WINNER "BEST RETRO GAME" 2010 BEST APP EVER AWARDS (iPad version)** The classic dungeon crawler adventure, now reimagined in a brand new adaptation for the Mac! Explore the perilous dungeon in your attemp


Platforms: Mac

The application is based on other open source technologies like POI, Lucene, Zkoss and Tagsoup. Key features "OSS Open Search Server": - Multi-language indexing. Documents can be indexed in ten languages OCo English, French, German, Spanish, Italian, Portuguese, Finnish, etc. - Multi-lingual...


Platforms: Windows, Mac, *nix, Java, BSD Solaris

The CMS - Bandits is a set of PHP scripts that implement an online HTML editor, calendar, search engine, RSS reader and editor, image gallery, comment system,Web crawler and many more.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Log RSS Creator allows you to watch live how users enter and leave your site.What information the Log RSS Creator shows :- title entry of the feed is the host name;- description entry is the list of pages visited by this host;- link entry is the link to the summary view for this host: which pages...


Platforms: PHP

This module ties into Feeds and the XPath Parser modules, allowing you to scrape a site's old content. You simply provide an XPath query for the next button on the page.Currently usable, yay!


Platforms: PHP

The GeoBlocker module allows site maintainers to black-list or to white-list access to a Drupal site by countries. IP tables come from http://www.php.net/source.php?url=/include/ip-to-country.inc and are downloaded locally to speed up lookup.Features * Supports blacklisting as well white-listing;...


Platforms: PHP

Marmoset's purpose is to provide a simple way to make your Web site's contents available through intelligent search.This module integrates the Calais Marmoset site crawler into Drupal. It adds a tab on the node that when selected will display the microformat data returned my marmoset.Click...


Platforms: PHP

PK-Torrents is a PHP torrent lister based on torrenthoster v1.0. It has the functionality to crawl the top torrent sites, Meganova, Mininova, Piratebay, Snarf, Torrentportal, Torrentspy.


Platforms: PHP

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own breed of DMOZ and Google.ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited number of links and pages without any loss of...


Platforms: PHP

RapidBot is the incredibly easy tool designed to create a robots.txt file for your site directly from RapidWeaver. Robots.txt file is retrieved by search engines, like Google and Bing, and used to state what pages need indexing and what must be ignored. A robot (also known as Spider or Web...


Platforms: Mac

PHP Search is a search engine script that searches a MySQL database for links and descriptions much like google. Manual adding of Data. Crawler Coming soon! Demo at http://www.jhosting.tk/admin/search/search.php


Platforms: *nix

This simple app does validation on all the HTML generated by your Django app, and keeps track of any pages that fail. You will no longer have any excuse for pages that aren't valid HTML What it does It uses a middleware to intercept all outgoing HTML and validate it using an external command...


Platforms: *nix

Content Grabber is a superior web scraping and web automation tool. It is very powerful, yet extremely easy to use. It can extract content from almost any website and save it as structured data in a format of your choice, including Excel reports, XML, CSV and most databases. Content Grabber...


Platforms: Windows, Windows 8, Windows 7, Windows Server

Complete crawler tool for SEO website audits including content, internal linking and broken references. Can also create all kinds of sitemaps including XML with hreflang, video, image, RSS/ROR, text, custom HTML/CSS/Javascript, DOT for visualization graphs and more. No fixed upper limit for...


Platforms: Windows, Windows 8, Windows 7, Windows Server

SiteAnalyzer it is a web crawler to scan the websites and check their technical and of SEO-parameters for errors and correct them effectively. Key features of the SiteAnalyzer - Scanning of all pages of the site, as well as images, scripts and documents - Getting server response codes for...


Platforms: Windows, Windows 8, Windows 7, Windows Server