RoboGen Dialogs

• Legacy Software
• RoboGen
• Help
• Dialogs

Dialogs

• Add Dialog
• Advanced Add Dialog
• Source Editor
• Header Editor
• FTP Login Dialog
• Robot Manager

Add Dialog Overview

The Add Path dialog is used to enter paths and files that you wish to disallow without logging onto your FTP server to select the file. You simply type in the path to the file or directory you want to disallow relative to your root. For example, if you had a directory for files that are constantly changing, such as www.mydomain.com/news/today then you would type /news/today1in the edit field and click OK or press Enter to add that directory to the disallow list.

If you want to add more than one path to the current agent’s disallow list at one time, choose the Advanced button. It will open the Advanced Add Dialog which will let you add any number of paths at one time.

1Note: It does not make sense for search engine robots to index constantly changing content since it will quickly become obsolete and only add to the huge amount of bad information returned by web searches.

↑ Back to Top

Advanced Add Dialog

The Advanced Add dialog is used to add more than one path to a user agent’s disallow list at one time. To add paths simply type them in the text box separated by lines breaks. Only one path can exist on any line of the text box. When you click OK, all the paths will be added to the disallow list.

To access the Advanced Add dialog, select the Advanced button in the Add Path dialog.

↑ Back to Top

Source Editor

The source editor dialog is used to preview and edit the robot exclusion source generated by RoboGen before it is saved. It is accessed by choosing Verify Source from the File menu.

The robot exclusion syntax is fairly straight forward. Each section has a user-agent tag and the associated disallow tags and comments are inserted in the file by preceding pound signs (#). The generated source for a simple robot exclusion file might look something like this:

1
2
3
4
5
6
7
8
9
10

# Rietta Solutions, Inc.  -- https://rietta.com
# Robot Exclusion File -- robots.txt
# Author:  Frank Rietta
# Last Updated:  05/23/99 21:27:40

User-agent: *
Disallow: /cgi-bin/
Disallow: /temp/        # these files change a lot
Disallow: /search.html  # search engines don't need to index this
Disallow: /about.html   # same as above

This exclusion code is fairly straight forward; it consists of two major parts, the optional header which is the group of comments at the top of the file and the disallow section. Notice that an asterisk (*) is used as the user-agent — that means that this disallow section will apply to all robots.

After you have made all the changes you want to make, simply click the Save As button to save the file.

Note: Even though RoboGen lets you save your exclusion file under any name on your computer, it must be called robots.txt when uploaded to you server.

↑ Back to Top

Header Editor

The header editor is used to edit your default header which is automatically placed at the top of all the exclusion files generated for you. If the header is left blank then RoboGen simply inserts “# robots.txt” at the top of your files. Each line of the header should begin with the comment tag, pound sign (#).

You can use the time stamp tag, —-timestamp—– to denote a location which the current time (retrieved from the system clock at save time) should be inserted into the header portion of your exclusion file.

Command Buttons

• Save – Saves the current header to the configuration file.
• Close – Closes the Header Editor.

↑ Back to Top

FTP Login Dialog

The Ftp Login dialog is used to collect the information that is required to log onto your web site’s FTP server to select the documents and directories to disallow. Once the login information is collected, choose the OK button and RoboGen will log onto the server.

Logging In

You enter the url or IP address to the site you are logging into in the FTP Server box, for example if your web site is www.mysite.com then your ftp server url could be ftp.mysite.com, please check with your web host if you are not sure what your FTP url is. If you need to use a user name and password to access your FTP site then check the This server requires me to log in check box and enter the information in the now enabled edit fields.

You can choose to store your password by selecting the Remember Password option. This is provided as a convenience and it might be a good idea to leave it unsolicited if others have access to your computer.

Document Root

The document root options let you specify the way you want RoboGen to handle the document root. You can let RoboGen auto detect common roots such as public_html, you can use the root that the server returns when logging on to it, or you can specify a directory to be the document root.

Create New Robot Exclusion File

If this option is selected then the current bot rules are cleared and a new a new workspace is created, otherwise all the current settings are left and you can continue to add more disallows to the same file.

↑ Back to Top

Robot Manager

The Robot Manager dialog is used to view and edit the information in the robot database files.

The window consists of four major parts, the Active Bot List is used to select which robot database is currently being viewed, the Listed Bots box list all the robots in the current database, the Robot Information frame displays the information for the currently selected robot, and the filters allow you to select which information is displayed in the Listed Bots list.

↑ Back to Top

Command Buttons

• Clear – Deletes all robot information in the current database.
• Remove – Deletes the currently selected robot from the current database.
• New Bot – Opens the New Robot dialog and allows you to add a robot to the current database.
• Modify – Allows you to edit the bot information for the current robot.

Goto Web Site (not shown) – when the currently displayed robot has a web site associated with it, this button opens that web page in your default web browser.

Close – Closes the Robot Manager.

↑ Back to Top