Crawler

DRKSpider will navigate a web site from a given URL, checking internal (and external) links, style sheets, images and other files. Broken links will be reported and the site map displayed in a tree control. Features: * Navigation tree matches site's navigation. * Detailed...

Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.

Fresh WebSuction is designed for downloading Web pages or web contents (reference material, software files, online books, e-zines, news articles) for viewing offline, and saving money on Internet connection fees. If you pay fees for the amount of time you are online you can save money by quickly...


Platforms: Windows

Power Search can find or capture anything embedded in HTML, CSS or JavaScript. Many websites are built from server-side scripts that dynamically generate content each time a page is loaded. Website content may come from databases, files, or from a different server all together. Power Search...


Platforms: Windows, 2003, Windows Vista, 2008, 7, 7x64

XenoCrawler is an application that allows you to crawl websites, servers, and web pages to view source code, style sheets, scripts, html, graphics, documents, and other related information safely, quickly, and easily without opening a browser or running any scripts. XenoCrawler is targeted for...


Platforms: Windows, Other

Netifera is a new modular open source platform for creating network security tools. This project provides many advantages for both security developers and researchers who want to implement new tools as well as the community of users of these tools. * TCP and UDP network scanning and service...


Platforms: Mac, Linux

Scrape websites and extract data into customizable CSV files ready to be imported anywhere like web databases and office tools. Also great for grabbing inventory lists you want to link or sell (e.g. if you are an affiliate). Mix and mash scraped website data to create new mashup website services...


Platforms: Windows

Are you getting too much junk email from your web site? Provide your email addresses to legitimate visitors of your web site without ending up on more junk email lists. Do it - fast and easy - with EScrambler. The entire program and all the documentation is on one web page. It doesnot get much...


Platforms: Windows, Mac, Linux,Palm,OS

Audit your website security with Acunetix Web Vulnerability Scanner As many as 70% of web sites have vulnerabilities that could lead to the theft of sensitive corporate data such as credit card information and customer lists. Hackers are concentrating their efforts on web-based applications -...


Platforms: Windows

Crawl-By-Example runs a crawl, which classifies the processed pages by subjects and finds the best pages according to examples provided by the operator. Crawl-By-Example is a plugin to the Heritrix crawler, and was done as a part of GSoC06 program.


Platforms: Windows, Mac, Linux

The little crane that could... The iOS hit makes its debut on the desktop, setting new standards for advanced physics simulation games. As Lisa Whitt puts it in her review: `It is creative apps like this that keep me on the mac app store not bad ports of old pc games.' Exclusive for this...


Platforms: Mac

It can be used in SEO tasks, hosting operations, tool optimizations, adding features to a project and many more.Bot Lane is a tabbed interface, each tab for a script that check an aspect of the site or contribute to its content.If a script is added, a tab will automatically be created....


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Node, taxonomy, and user information are all available out of the box, with the option to name the selected data however needed. Any data can be combined into a comma-separated list as a single Metatag as well. Individual settings are stored for each content type, plus default settings that can...


Platforms: BSD Solaris

This contribution exports the shopboy.txt for your shop to the ShopBoy Price comparison Crawler.For german shops only.


Platforms: Windows, *nix, PHP, BSD

Node, taxonomy, and user information are all available out of the box, with the option to name the selected data however needed. Any data can be combined into a comma-separated list as a single Metatag as well. Individual settings are stored for each content type, plus default settings that can...


Platforms: BSD Solaris

Maian Search is a very simple search engine for your website that stores your pages in a database. It is ideal for small websites that don`t need a site crawler to regularly update pages. Easily create a custom search page and an import function is also included. Features: - Simple search engine....


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Sitemap Creator is a php script making use of the announced new standard SiteMaps protocol supported by Google, Yahoo and MSN and pings them to crawl it. It also sends an email to you with the crawler info and which sitemap was crawled.


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

This blocks e-mail address mining robots and spam crawlers.InstallationUnpack and upload it to the /wp-content/plugins/ directory.Activate the plugin through the 'Plugins' menu in WordPress. Requirements: - WordPress 2.0 or higher


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

On WP-MU installations it allows for every blog to have its own robots.txt file.InstallationUnpack and upload it to the /wp-content/plugins/ directory.Activate the plugin through the 'Plugins' menu in WordPress. Requirements: - WordPress 2.0 up to 2.5


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

It can help create large sites consisting of many pages where a consistent look, navigation and layout is necessary.The application intercepts requests to any static or dynamically generated HTML page requested through the web-server, parses the page, obtains properties and data from the content...


Platforms: Java