This glossary is intended to help provide greater explanation of the terms used in this documentation. The author is avoiding redefining common terms of which numerous definitions may be found. It is assumed that the reader is familiar with terms such as FTP, ASCII, and directories.
The document root is the server path that is accessible by web visitors and robots. While it can be the server root, it is commonly set as a secondary directory such as public_html. There is nothing special about this name and the directory which it is set varies from web server to web server. Please consult with your systems administrator, virtual domain host, or server documentation if you are not sure what your document root is.
RoboGen Specific – it is the list on the left of the main window in which agent rules are viewed for the currently selected robot.
Robot Exclusion Protocol
The standard used for robot exclusion files. It defines the syntax and location for ROBOTS.TXT and how web robots are to parse that file. Each robot has a user-agent, which is its handle, and must follow all directives under the section for its user-agent. If there are no directives specific to its user-agent, then the robot is to follow all directives under the universal user-agent (which is denoted by an asterisk). Also, the robot exclusion file must be called ROBOTS.TXT and reside in the server’s document root.
The file name utilized by the robot exclusion protocol. Web robots download this file from the server’s document root and parse it for instructions on what to index and not to index. The case of the file name does not matter, but it must exist in the document root.
Also known as a Web Wanderer or Web Spider, it is a program that traverses the Internet automatically by retrieving a document, and recursively retrieving all documents that are referenced. Robots can perform any number of functions, but the most common uses are: indexing, validating HTML, validating links, “What’s New” monitoring, and site mirroring.
© 2000 Rietta. All Rights Reserved.
This HTML file is part of the documentation for Rietta Solutions’s RoboGen application. Click here for more information about RoboGen.