Introduction
You see here part of a special document in the CMS that is used to configure the response of the site to robots. This is made possible by the robotstxt addon.
What is robotstxt?
This Hippo Site Toolkit component allows webmasters to configure the robots.txt response for their websites.
It contains a document type to configure which paths are allowed and front-end logic to return its content
in the correct format.
The demo sub project contains content, HST configuration and a JSP file to render the robots.txt
System requirements
The latest version of robotstxt is 1.03.00. It runs with Hippo Site Toolkit 2.04.xx and up and Hippo CMS 7.4 (2.16.xx).
Project status and plans
For a basic robots.txt the project is ready for production.
The following items are supported:
- User-agent
- Disallow
See the project tracker for a list of bugs and feature requests.
Where is the demo?
See the demo project.
How can I use robotstxt in my project?
Follow the installation instructions!