Crawlers

ETagFix is an ISAPI plugin for Internet Information Server (IIS) that changes the etag produced by IIS so that your etag does not change every time your computer reboots. This can result in web crawlers and user clients wanting to recache your web page when they don't need to. Platforms: Windows

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once! Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to... Platforms: Windows, *nix

Mod_Spambot is an Apache module which monitors the data being downloaded from a server. When the number of requests for a client exceeds a preset level no more downloads are allowed for a preset time. When this happens the client received a tailored message informing them of what has happend.... Platforms: *nix

jGetFile project is geared towards mass downloading specifically non-html files from the web. Web crawlers and website downloads are widely available and work very well. Not all web-based file crawlers are equal however. Few file crawers handle this href scenario: < a... Platforms: *nix

IndexFinger is a bare-bones program with one simple function: creating an index of every web page in a folder on your hard drive. It takes the amazingly boring task of finding every html page in every folder and sub-folder and creating one page which links directly to each of those pages. Who... Platforms: Mac

Xfilesharing Free gives you ability to run your own file sharing site like Rapidshare or MegaUpload. Absolutely for free. Can be installed on any virtual/shared or dedicated hosting running Perl (most servers do). You can customize look and feel of your site with just editing HTML files and... Platforms: Windows, Mac, *nix, Source

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once! Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to... Platforms: Windows, *nix

The web crawler is a program that automatically traverses the web by downloading the pages and following the links from page to page. A general purpose of web crawler is to download any web page that can be accessed through the links. This process is called web crawling or spidering. Many sites,... Platforms: Windows, Mac, Linux, Linux Console, Linux Gnome, Linux GPL, Linux Open Source, Java

In the blink of an eye you an overview of the activity on your website. The program generates statistics in real time, you also have access to the historical past months. Access to your site are broken, a section for visitors (humans), a section for robots or crawlers that scan the site. Whether... Platforms: All

The main features: Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards. Simulation crawling web crawlers,can check out the incorrect link. Set up a free multi-threaded, to generate extremely fast. Manual support Web site editor, artificial setting... Platforms: Windows

The main features: Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards. Simulation crawling web crawlers,can check out the incorrect link. Set up a free multi-threaded, to generate extremely fast. Manual support Web site editor, artificial setting... Platforms: Windows

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen... Platforms: Windows

The main features: Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards. Simulation crawling web crawlers,can check out the incorrect link. Set up a free multi-threaded, to generate extremely fast. Manual support Web site editor, artificial setting... Platforms: Windows

The main features: Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards. Simulation crawling web crawlers,can check out the incorrect link. Set up a free multi-threaded, to generate extremely fast. Manual support Web site editor, artificial setting... Platforms: Windows

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud. The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites. D-Guard Web Application Firewall is designed... Platforms: Windows

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud. The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites. D-Guard Web Application Firewall is designed... Platforms: Windows

Are you getting too much junk email from your web site? Provide your email addresses to legitimate visitors of your web site without ending up on more junk email lists. Do it - fast and easy - with EScrambler. The entire program and all the documentation is on one web page. It doesnot get much... Platforms: Windows, Mac, Linux,Palm,OS

An ultimate 25 game pack of arcade games including arkanoid and breakout style games, pacman style games and fast and franctic space shooters. Incredible music, sound and graphics with many configurable options in each of the titles. Most games include features such as keyboard, mouse or... Platforms: Windows

The Java Sitemap Parser can parse a website's Sitemap (http://www.sitemaps.org/). This is useful for web crawlers that want to discover URLs from a website that is using the Sitemap Protocol. Platforms: Windows, Mac, Linux

Open Search Server (OSS) is a search engine software developed under the GPL v3 open source licence. Built using the best open source technologies available, Open Search Server is a stable, high-performance piece of software. It is both a modern search engine and a suite of high-powered full text... Platforms: Mac