Robotstxt

This module generates the robots.txt file dynamically and gives you the chance to edit it, on a per-site basis, from the web UI.Installation - Unpack to your Drupal site modules directory (usually /sites/all/modules/) - Enable from Administer > Site Building > Modules  Requirements: - Drupal...


Platforms: Windows, Mac, *nix, PHP, BSD Solaris

TracRobotsTxt is a module that will serve a robots.txt file from Trac. Mostly useful to tracd users, but works on anything. Just put the data you want in the wiki page RobotsTxt. Configuration: To enable the plugin: [components] robotstxt.* = enabled A typical RobotsTxt wiki page will look...


Platforms: *nix

This is a simple and lightweight content management system for creating dynamic websites. Can be used for building blogs, wikis, ticket systems, documentation sites, or just plain old static webistes.Features of LitePublisher:- Image gallery- Audio player- Video player- Media RSS- RSS- Plugins-...


Platforms: PHP

LinkChecker script allows you to check websites and HTML documents for broken links.Features: - recursive checking- multithreaded- output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats- HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Gopher, Telnet and local...


Platforms: Python

A Python library for scraping things. Features include: * HTTP, HTTPS, FTP requests via an identical API * HTTP caching, compression and cookies * redirect following * request throttling * robots.txt compliance (optional) * robust error handling scrapelib is a project of Sunlight Labs (c)...


Platforms: *nix

SEOpen is an extension I made for Firefox to help with search engine optimization. It provides many helpful tools at the click of a mouse. SEOpen features are accessed through right-clicking an open area on a web page or by using the included toolbar; they include: * Yahoo Backlinks * Yahoo...


Platforms: *nix

pyramid_assetviews is a small module that allows you to add Root-Relative views to your project. This enables you to serve assets that need to be accessed from the root of your domain such as robots.txt or favicon.ico without having to setup a static media server such as nginx or lighttpd.


Platforms: *nix