Visual Web Spider

ItSucks software is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates. The application also provides a swing GUI and a console interface. All backend functionalities are also available... Platforms: *nix

Fetchgals is a Web spider that locates and optionally downloads free pictures and movies from the internet. Today, commercial porn sites are advertised like this: somebody puts together a thumbnail gallery, i.e. a web page with small thumbnail pictures linking to free full-size images or... Platforms: *nix

Sphider is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is suitable for adding search functionality to small or medium sites (up to around 100,000 pages). It also works great as a tool for site analysis - finding broken links, gathering... Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Visual web page creator. Uses wizards, drag 'n drop. Easy to use, eg. wysiwyg editing, templates, comprehensive help, single-click upload. Advanced features, eg code editing, automated download of existing web pages, HTML 4.0 support. Includes fonts, GIFs, scripts, page backgrounds, etc. Award... Platforms: Windows

Inquisitor is a very fast & save visual Web Picture Site Ripper, Downloader/Grabber, and Offline Browser. "Visual" means that it works like a normal Browser, but with "WYSIWYG" (WhatYouSeeIsWhatYouGet) principle to download all files/images at once. Platforms: *nix

IRobot is the Internet-robot engine for Visual Web automation and Web scraping. Completely automatic Web recording. Create your own IRobot Web robot agents to automate everything on the Web. Support Web data integration and computation. Share your Web experience with your friends. Batch form... Platforms: Windows

PhpDig is a web spider and PHP search engine project, using a MySQL database and flat file support.PhpDig search engine can store textual content of indexed documents in files.PhpDig search engine builds a glossary with words found in indexed pages. On a search query, it displays a result page... Platforms: PHP

A simple web spider with several recursion strategies. Home page is at http://github.com/slinkp/spydey. It doesn't do much except follow links and report status. I mostly use it for quick and dirty smoke testing and link checking. The only unusual feature is the ``--traversal=pattern`` option,... Platforms: *nix

Visual SEO Studio is a desktop SEO Audit Tool, a Visual approach to make SEO tasks quick and easy. The program lets you Crawl one or more sites in parallel with its Interactive Web Spider, Spot on-site issues at a glance, Save Page Screenshot miniatures and see where the fold lays, visually... Platforms: Windows, Windows 7, Windows Server

Spider Tracker is a WordPress plugin developed by Tony Fernandes of MnM-Designs.com with one purpose. That purpose is to track web spiders' activity on your WordPress blog. It features a friendly user interface allowing you to define which spiders to track and how many recent index crawls to... Platforms: PHP

The Microsoft Silverlight Tools for Visual Studio package is an add-on for Visual Studio 2010 to provide tooling for Microsoft Silverlight 5 Beta and Microsoft WCF RIA Services V1.0 SP2 Preview (April 2011). It can be installed on top of either Visual Studio 2010 Service Pack 1 or Visual Web... Platforms: Windows, Windows Vista, 2008, 7

InstantLogic is a Windows-based, visual, high-productivity development environment for creating and deploying data-driven Web applications. InstantLogic automatically handles the optimizations and routine tasks that have become time-wasters in modern, data-driven Web development. With... Platforms: Windows

SiteOrbiter is a visual reporting tool for web developers and administrators. It enables the user to graphically report on broken links, see link relationships in a visual map, and get a quick understanding of a websites complexity. Get familiar with a web site quickly and easily. With the... Platforms: Mac

The Microsoft Web Platform Installer was designed to be a free tool that makes it simple to download and install the latest components of the Microsoft Web Platform, including the IIS7, SQL Server Express, .NET Framework and Visual Web Developer. Developers and server administrators can use... Platforms: Windows

The Microsoft Web Platform Installer was designed to be a free tool that makes it simple to download and install the latest components of the Microsoft Web Platform, including the IIS7, SQL Server Express, .NET Framework and Visual Web Developer. Developers and server administrators can use... Platforms: Windows

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own little breed of DMOZ and Google. ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited ammount of links and pages. Powered by... Platforms: Windows, Mac, *nix, PHP, BSD Solaris

It is basically a program that can make you a search engine. It is a web crawler, has all the web site source code (in ASP, soon to be PHP as well), and a mysql database. Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since this crawler seeks to... Platforms: Windows, Mac, *nix, Java, BSD Solaris

Larbin is a Web crawler intended to fetch a large number of Web pages. It should be able to fetch more than 100 millions pages on a standard PC with much u/d. This set of PHP and Perl scripts, called webtools4larbin, can handle the output of Larbin. Platforms: Windows, Mac, *nix, PHP, BSD Solaris

WWW::PkgFind watches Web sites, FTP sites, GIT repositories, etc. for new code releases, and downloads them. In other words, its like a Web spider tuned for downloading software packages and patches. WWW::PkgFind is also able to generate a queue of incoming packages, to allow subsequent... Platforms: *nix