American Funnel Web Spider

Spider websites to find slow or broken links, view SEO meta-data and measure performance. SiteScan is a Windows desktop utility to spider websites for slow and broken links (including the HTTP response 404 not found error). It quickly crawls and validates all web pages, images, file...

Aggregate, manipulate, and mashup data in one simple tool. Djuggler (short for data juggler) is a hands-on visual scripting tool to mashup and collect data from multiple sources like web pages, text files, data grids as Microsoft Excel or CSV files. When the data has been collected or...

This superb program allows you to automatically download pictures, movies, mp3 and other files to your hard drive for offline viewing. Itls a highly configurable, multi-threaded, file-retrieving Web spider. It's perfect for automated researching and downloading files from Web sites. It...


Platforms: Windows

"Blindingly fast" (Computer Forensics:Incident Response Essentials) searching of gigabytes of word processor, database, spreadsheet, ZIP, XML, PDF, HTML, emails and more. *Over two dozen indexed, unindexed, fielded and full-text search options. *Highlights hits in retrieved files; for HTML & PDF,...


Platforms: Windows

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since this crawler seeks to...


Platforms: Windows, Mac, *nix, Java, BSD Solaris

Larbin is a Web crawler intended to fetch a large number of Web pages. It should be able to fetch more than 100 millions pages on a standard PC with much u/d. This set of PHP and Perl scripts, called webtools4larbin, can handle the output of Larbin.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Fast, detailed website analysis for web-masters and content managers. Find slow, broken or mistyped links. Gather page and file meta-data. Find referrers and server redirects. Extract email addresses. Measure response and browser load-time. Save and export results. Check any type of web page,...


Platforms: Windows, Windows 8, Windows 7, Windows Server

InterroBot combines a web crawler with advanced filtering to help you get answers to just what the heck is going on with your website. Whether you're looking for a snippet of text in the body of your target webpages, the HTTP headers, or a specific HTTP status code-InterroBot will help you...


Platforms: Windows, Windows 8, Windows 7, Windows Server

WWW::PkgFind watches Web sites, FTP sites, GIT repositories, etc. for new code releases, and downloads them. In other words, its like a Web spider tuned for downloading software packages and patches. WWW::PkgFind is also able to generate a queue of incoming packages, to allow subsequent...


Platforms: *nix

Visual SEO Studio is a desktop SEO Audit Tool, a Visual approach to make SEO tasks quick and easy. The program lets you Crawl one or more sites in parallel with its Interactive Web Spider, Spot on-site issues at a glance, Save Page Screenshot miniatures and see where the fold lays, visually...


Platforms: Windows, Windows 7, Windows Server

Webskape Browser is a web browser for public use that can be locked down to make browsing the internet secure and the operating system tamperproof. Great for cyber cafes, libraries, malls, airports, shops, museums, art galleries and more... Internet Explorer Webskape Browser uses the same...


Platforms: Windows

MailsExtractor is an easy-to-use and fast email collection tool. It works like any website spider. It can open all the pages from a site, and it will collect all the emails it finds. You should only setup a task with serval options at first, and then you can start or pause it whenever you want....


Platforms: Windows

About NetScrape If you like images on the web, you ll like NetScrape. It is a page-sucking web spider, surgically attached to an "aqua-licious" image thumbnailer and viewer. In this regard, it s a lot like Frankenstein - and he s pretty cool, right? Give NetScrape a URL, and it will "scrape" all...


Platforms: Mac

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own breed of DMOZ and Google.ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited number of links and pages without any loss of...


Platforms: PHP

Pattern is a web mining module for the Python programming language. It bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine...


Platforms: *nix

SiteAnalyzer it is a web crawler to scan the websites and check their technical and of SEO-parameters for errors and correct them effectively. Key features of the SiteAnalyzer - Scanning of all pages of the site, as well as images, scripts and documents - Getting server response codes for...


Platforms: Windows, Windows 8, Windows 7, Windows Server

Visual SEO Studio is a desktop SEO Audit Tool, a Visual approach to make SEO tasks quick and easy. The program lets you Crawl one or more sites in parallel with its Interactive Web Spider, Spot on-site issues at a glance, Save Page Screenshot miniatures and see where the fold lays, visually edit...


Platforms: Mac

SXBandMaster is designed for Internet services maximum use. It is a program complex that serves as ISP Dialer, Download Manager, Offline Browser (Web Spider), FTP Upload Application with FXP support, Time Synchronizer (Atomic Clock), Traffic Explorer and many others. SXBandMaster excels it's...


Platforms: Windows

SXBandMaster is designed for Internet services maximum use. It is a program complex that serves as ISP Dialer, Download Manager, Offline Browser (Web Spider), FTP Upload Application with FXP support, Time Synchronizer (Atomic Clock), Traffic Explorer and many others. SXBandMaster excels it's...


Platforms: Windows

PHP FastFileCache caches output from dynamic PHP scripts, and stores them in files for fast retrieval under high server load.. WebSuck is a web spider for Java2 (C) 2001 Åke Wallebom


Platforms: *nix