Make Robots Txt

On WP-MU installations it allows for every blog to have its own robots.txt file.InstallationUnpack and upload it to the /wp-content/plugins/ directory.Activate the plugin through the 'Plugins' menu in WordPress. Requirements: - WordPress 2.0 up to 2.5

A robots.txt file lets search engines (Google, Yahoo, MSN, etc) know which pages on your site you don't want them to index. This is helpful in keeping them from going to pages that will confuse them and/or won't provide any added benefit to you from having indexed.Since nobody has put one in here...

Visual SEO Studio is a desktop SEO Audit Tool, a Visual approach to make SEO tasks quick and easy. The program lets you Crawl one or more sites in parallel with its Interactive Web Spider, Spot on-site issues at a glance, Save Page Screenshot miniatures and see where the fold lays, visually...


Platforms: Windows, Windows 7, Windows Server

This module generates the robots.txt file dynamically and gives you the chance to edit it, on a per-site basis, from the web UI.Installation - Unpack to your Drupal site modules directory (usually /sites/all/modules/) - Enable from Administer > Site Building > Modules  Requirements: - Drupal...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Installation - Unpack in your modules folder (usually /sites/all/modules/) and enable under Administer > Site Building > Modules. Requirements: - Drupal 6.x


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

RobotPack is a free robot-exclusion management tool to inform robots and other automated search engine tools not to index parts of your Web site. RobotPack provides an FTP access to your Web site so you can easily select files and directories which you don't want indexed by search engines....


Platforms: Windows

The Revisionist is a tool for extracting and indexing hidden metadata (such as deleted or modified text) from large collections of MS Word files. It can operate whole Web sites or SMB or NFS directories. The Revisionist project is handy for pen-testing, or it can be used just to spot...


Platforms: *nix

This simple robots.txt file will help deter some of the unwanted bots that spider your site for email adresses and things of that nature that result in spam. This works for all websites and it is not just limited to phpBB.This has also been configured to allow google adsense bots to spider your...


Platforms: Windows, *nix, PHP, BSD

TracRobotsTxt is a module that will serve a robots.txt file from Trac. Mostly useful to tracd users, but works on anything. Just put the data you want in the wiki page RobotsTxt. Configuration: To enable the plugin: [components] robotstxt.* = enabled A typical RobotsTxt wiki page will look...


Platforms: *nix

Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.


Platforms: Windows

Crawl a website with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.


Platforms: Windows

LinkChecker is a simple, very easy to use application specially designed to help you check websites and HTML documents for broken links. You just enter the URL or HTML doc to check, press Start and let LinkChecker take care of the rest. Features: recursive and multithreaded checking output in...


Platforms: Windows

LinkChecker is a simple, very easy to use application specially designed to help you check websites and HTML documents for broken links. You just enter the URL or HTML doc to check, press Start and let LinkChecker take care of the rest. Features: recursive and multithreaded checking output in...


Platforms: Windows

LinkChecker is a simple, very easy to use application specially designed to help you check websites and HTML documents for broken links. You just enter the URL or HTML doc to check, press Start and let LinkChecker take care of the rest. Features: recursive and multithreaded checking...


Platforms: Linux

LinkChecker is a simple, very easy to use application specially designed to help you check websites and HTML documents for broken links. You just enter the URL or HTML doc to check, press Start and let LinkChecker take care of the rest. Features: recursive and multithreaded checking...


Platforms: Mac

Sphider is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is suitable for adding search functionality to small or medium sites (up to around 100,000 pages). It also works great as a tool for site analysis - finding broken links, gathering...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Source code can be entered with for example ... or ....Supports highlighting for the following languages: ABAP, ActionScript, Ada, Apache Log, AppleScript, ASM, ASP, AutoIt, Bash, BlitzBasic, bnf, C, C (Mac), CAD DCL, CAD Lisp, CFDG, ColdFusion, C , C (QT), C#, CSS, D, Delphi, Diff, DIV, DOS,...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since this crawler seeks to...


Platforms: Windows, Mac, *nix, Java, BSD Solaris

This is a simple and lightweight content management system for creating dynamic websites. Can be used for building blogs, wikis, ticket systems, documentation sites, or just plain old static webistes.Features of LitePublisher:- Image gallery- Audio player- Video player- Media RSS- RSS- Plugins-...


Platforms: PHP

Adds three more levels of privacy to the Options--Privacy page. 1. Blog visible to any logged in community member - "Network Users Only". 2. Blog visible only to registered users of blog - "Blog Members Only". 3. Blog visible only to administrators - "Admins Only".Mulitsite SuperAdmin can set an...


Platforms: PHP