Copart Crawler

Marmoset's purpose is to provide a simple way to make your Web site's contents available through intelligent search.This module integrates the Calais Marmoset site crawler into Drupal. It adds a tab on the node that when selected will display the microformat data returned my marmoset.Click... Platforms: PHP

JCrawler is a perfect cralwing/load-testing tool which is cookie-enabled and follows human crawling pattern (hit/second). Platforms: Windows, Mac, Linux

The GeoBlocker module allows site maintainers to black-list or to white-list access to a Drupal site by countries. IP tables come from http://www.php.net/source.php?url=/include/ip-to-country.inc and are downloaded locally to speed up lookup.Features * Supports blacklisting as well white-listing;... Platforms: PHP

a minimal Java web crawler Java web crawler Web Site Features http crawler validate domains Platforms: Mac

Bee-rain is a web crawler that harvest and index file over the network. It is a free PHP search engine software.Bee-rain helps you in your sleep by comparing, filtering, analyzing citations of brands, companies and personalities on the Internet. Bee-rain then produces statistics and visualization... Platforms: PHP

This module ties into Feeds and the XPath Parser modules, allowing you to scrape a site's old content. You simply provide an XPath query for the next button on the page.Currently usable, yay! Platforms: PHP

This is a Professional e-mail address verification tool.Rapid and accurate test-mail address is real and effective. Key Features: Up to 5,000 threads set free(Depend on your machine).Can verify about 150000 email per hours. Only verify email is valid(Real exist) is invalid and nothing is send to... Platforms: Windows, Linux

Log RSS Creator allows you to watch live how users enter and leave your site.What information the Log RSS Creator shows :- title entry of the feed is the host name;- description entry is the list of pages visited by this host;- link entry is the link to the summary view for this host: which pages... Platforms: PHP

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own breed of DMOZ and Google.ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited number of links and pages without any loss of... Platforms: PHP

PK-Torrents is a PHP torrent lister based on torrenthoster v1.0. It has the functionality to crawl the top torrent sites, Meganova, Mininova, Piratebay, Snarf, Torrentportal, Torrentspy. Platforms: PHP

Complete crawler tool for SEO website audits including content, internal linking and broken references. Can also create all kinds of sitemaps including XML with hreflang, video, image, RSS/ROR, text, custom HTML/CSS/Javascript, DOT for visualization graphs and more. No fixed upper limit for... Platforms: Windows, Windows 8, Windows 7, Windows Server

Bitvo is a media crawler and downloader. Point it to a web site or thumbnail gallery and it will seek out images, videos, and audio files and download them directly to your computer. Platforms: Mac

AWSpider is a web crawler for Amazon Web Services, written in Python. #md5=53f2f7be2f926c9f4b2c867b657ef59b Platforms: Mac

RapidBot is the incredibly easy tool designed to create a robots.txt file for your site directly from RapidWeaver. Robots.txt file is retrieved by search engines, like Google and Bing, and used to state what pages need indexing and what must be ignored. A robot (also known as Spider or Web... Platforms: Mac

AWSpider is a web crawler for Amazon Web Services, written in Python. #md5=edb1228e36df1a72e28bb07119e7a2e2 Platforms: *nix

PHP Search is a search engine script that searches a MySQL database for links and descriptions much like google. Manual adding of Data. Crawler Coming soon! Demo at http://www.jhosting.tk/admin/search/search.php Platforms: *nix

Paglo Crawler discovers all devices connected to a network - including workstations, servers, switches, routers, printers, etc and gathers rich information about each device. This information is then searchable through an account at http://paglo.com/ Platforms: *nix

This simple app does validation on all the HTML generated by your Django app, and keeps track of any pages that fail. You will no longer have any excuse for pages that aren't valid HTML What it does It uses a middleware to intercept all outgoing HTML and validate it using an external command... Platforms: *nix

FoxySpider is your personal web crawler! It can crawl into a given web site and retrieve what you really want (video clips, images, music files, etc.). FoxySpider displays the located items in a well-structured thumbnail gallery for ease of use. Platforms: Mac

AWSpider is a web crawler for Amazon Web Services, written in Python #md5=edb1228e36df1a72e28bb07119e7a2e2 Platforms: *nix