RoboGen
Because RoboGen is discontinued, it is no longer available for purchase or support. If you would like to use the software, please use this complimentary license key.
To activate your copy, enter the following information:
- Name: Complimentary RoboGen License
- Company: Rietta Inc.
- Product ID: 24160-e472d
RoboGen is a program, for Windows 2000, XP, and Vista,to help you more easily manage the robots.txt files for your websites. Instead of of the time consuming manual management process, described in the tutorial, you can either edit your robot exclusion files on your local computer or on a remote FTP site.
RoboGen Information
- Introduction to Robot Exclusion Files and RoboGen’s Features
- Screen Shots of RoboGen
- Download RoboGen today
- Read RoboGen Documentation
- More Robots.txt Resources
Introduction to Web Robots, ROBOTS.TXT, and RoboGen
Search engines such as Google and Bing a use web spiders, also known as robots, to create the indexes for their search databases. These robots transverse HTML trees by loading pages and following hyperlinks, and they report the text and/or meta-tag information to create search indexes. ROBOTS.TXT, a file that spiders look in for information on how the site is to be cataloged. It is a ASCII text file that sits in the document root of the server. It defines what documents and/or directories that confirming spiders are forbidden to index.
The robot exclusion protocol was introduced by Martijn Koster in 1994 to deal with problems that had been arising due to the increasing popularity of the internet and the toll web spiders were having on system resources. Some of the problems were caused by robots rapid-firing requests, that is loading pages in rapid succession. Other problems such as robots indexing information deep in directory trees, temporary information, and even accessing cgi-scripts. The robot exclusion protocol was quickly adopted by webmasters and web robot makers as a way to organize and control the indexing process.
Since then, the size of the Internet has increased dramatically and millions of people are using it. The number of web robots crawling the web is greater than before and it is more important than ever for all web sites to have a properly created and maintained ROBOTS.TXT file.
With RoboGen you create robot exclusion files by selecting All Robots or a specific user-agent and adding documents and/or directories by entering the path names manually or by selecting them using FTP. Once all the restrictions and directives are set you can save the robots.txt file to your hard drive or upload it directly to your server.
It is important to remember that robot exclusion files are not a security measure. Some robots will simply ignore the file and others may purposely load the documents that the files marks as disallowed. This means that robot exclusion files are really only useful for controlling what appears in search engines.
RoboGen Product Overview
RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public. RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable.
Feature Overview
- Create Robot Exclusion Files by selecting documents and directories.
- Log into FTP servers and even upload robots.txt from RoboGen.
- Manage information for multiple servers.
- Database of over 180 know user-agents and 10 major search engines.
- Ability to edit the robot database and add additional user-agents.
- RoboGen recognizes common document roots when logging into FTP servers and suggests that they be set as the document root.
For a detailed overview of RoboGen’s features, see the RoboGen Feature Comparison Chart.
Screenshots
RoboGen has been designed to make managing the robot exclusion files which can control how search engines visit websites as easy as possible. Here are some screen shots of RoboGen in action.
Most of the operations of RoboGen center around the main window. (Click to Enlarge)
The Ftp Dialog is where you manage your FTP servers and passwords. It can store the information for multiple sites.
The Robot Manager provides is where you view and edit the user-agent information in the robot database. RoboGen comes with information on over 180 known web robots and the robot information for 10 of the most popular search engines.
The header editor is used to edit the header that is placed at the top of all the robot exclusion files you create.
The options dialog lets you set general FTP and help file options.
The source editor is used to edit the generated robot exclusion code before saving it to the disk.
If you wish to enter documents or directories manually, use the Add Relative Path dialog.
The Advanced Add dialog allows multiple documents and paths to be entered at once. (One per line.)
RoboTag is a small companion program used to create Robot Exclusion META tags. Click here for more information on RoboTag.
Download
RoboGen Standard Edition
You are only minutes away from being able to easily control how search engines interact with your web sites. If you are still not sure that the RoboGen product is right for you, please download the Trial (link to the right) and test it before placing your order.
RoboGen SE has been tested with Windows 2000, XP, and Vista.
Remember, as of January 2014, you can use the complimentary key to unlock RoboGen SE for free:
- Name: Complimentary RoboGen License
- Company: Rietta Inc.
- Product ID: 24160-e472d
RoboTag 1.5
This is the free companion product for RoboGen, it can be used to create Robot Exclusion Meta Tags to be inserted in your HTML pages. This program is already included with the main RoboGen product.