Web Spider

Visual Web Spider is a software for crawling the web. It is a web crawler. It helps you to develop search directory for responsible website promotion, or for research purposes. Visual Web Spider builds targeted URL database at high speed. Our program allows you to extract URLs and metadata...

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen...

There are many websites that you visit a lot for their content. The bad side is that you always need an Internet connection, but there is a solution to see them locally in your PC without needing an Internet connection.JOC Web Spider is a download accelerator that lets you download whole websites...


Platforms: Windows

ItSucks software is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates. The application also provides a swing GUI and a console interface. All backend functionalities are also available...


Platforms: *nix

Fetchgals is a Web spider that locates and optionally downloads free pictures and movies from the internet. Today, commercial porn sites are advertised like this: somebody puts together a thumbnail gallery, i.e. a web page with small thumbnail pictures linking to free full-size images or...


Platforms: *nix

Sphider is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is suitable for adding search functionality to small or medium sites (up to around 100,000 pages). It also works great as a tool for site analysis - finding broken links, gathering...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Use SiteScan to spider entire websites, find broken links, report server errors and analyze page response times. SiteScan will test every page on your website, list the results and highlight troublesome pages. Preview pages using the built-in browser, quickly highlight links (or images) on any...


Platforms: Windows

HTML Link Validator is a professional tool for checking web pages for broken links, on a web server or on your local computer. Whether a novice or an experienced webmaster, you should always test your web site for broken links. There are many reasons why a link may be broken. The file name may be...


Platforms: Windows

Hit Discovery is a multi-threaded web-spider designed to search for and download mp3 files that allows you to listen to mp3 previews before downloading them. It will search for mp3 files you want and download them from any website on the Internet. The main feature of Hit Discovery is that it...


Platforms: Windows

look4me is an email extractor, internet web spider, image extractor, fax extractor and email sender built upon industrial strength scanning technologies. It is extremely powerful, allowing you to extract useful information from the internet quickly and easily. look4me can be used to extract the...


Platforms: Windows

PhpDig is a web spider and PHP search engine project, using a MySQL database and flat file support.PhpDig search engine can store textual content of indexed documents in files.PhpDig search engine builds a glossary with words found in indexed pages. On a search query, it displays a result page...


Platforms: PHP

A simple web spider with several recursion strategies. Home page is at http://github.com/slinkp/spydey. It doesn't do much except follow links and report status. I mostly use it for quick and dirty smoke testing and link checking. The only unusual feature is the ``--traversal=pattern`` option,...


Platforms: *nix

dtSearch Reviews* "The most powerful document search tool on the market"-- Wired Magazine* "dtSearch ... leads the market" -- Network Computing* "Finding a virtual needle in a digital haystack is now much easier" -- MS OfficePRO* "Blindingly fast" -- Computer Forensics: Incident Response...


Platforms: Windows

Spider Tracker is a WordPress plugin developed by Tony Fernandes of MnM-Designs.com with one purpose. That purpose is to track web spiders' activity on your WordPress blog. It features a friendly user interface allowing you to define which spiders to track and how many recent index crawls to...


Platforms: PHP

Custo is a fully automated, multithreaded, file-retrieving Web spider. The program lets you explore a Web site's resources ordered in a file-tree structure, verify the accessibility of the links to other sites, and generate a list of these links. You can then explore, download, and open selected...


Platforms: Windows

The leading Personal Web Automation Suite, Web Scraper Plus+ makes it possible to login to a secure website --> submit a search form --> crawl the results --> and scrape sections and fields of a resulting html pages to rows and columns in your favorite spreadsheet or database. It is a fusion of a...


Platforms: Windows

Astanda Directory Project [ADP] is a directory, search engine, and spider/crawler software. Your own little breed of DMOZ and Google. ADP is the only software in its class ever to be written in PHP with MySQL database support. Supports virtually unlimited ammount of links and pages. Powered by...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

It is basically a program that can make you a search engine. It is a web crawler, has all the web site source code (in ASP, soon to be PHP as well), and a mysql database.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Spider websites to find slow or broken links, view SEO meta-data and measure performance. SiteScan is a Windows desktop utility to spider websites for slow and broken links (including the HTTP response 404 not found error). It quickly crawls and validates all web pages, images, file...


Platforms: Windows 8, Windows 7, Windows Server

Aggregate, manipulate, and mashup data in one simple tool. Djuggler (short for data juggler) is a hands-on visual scripting tool to mashup and collect data from multiple sources like web pages, text files, data grids as Microsoft Excel or CSV files. When the data has been collected or...


Platforms: Windows