https://rietta.com/robogen/ Recent content in RoboGen on Rietta Cybersecurity Hugo -- gohugo.io en-us 1999-2026 Rietta Inc. All Rights Reserved. https://rietta.com/robogen/moreinfo/ Mon, 01 Jan 0001 00:00:00 +0000 https://rietta.com/robogen/moreinfo/ Additional Resources This product is legacy software that is no longer maintained, nor supported by Rietta Inc. This page is preserved for historical purposes. See the listing of Rietta’s legacy software for a complete list of this software. General ROBOTS.TXT Information Martijn Koster’s site about the Robot Exclusion Protocol. This is the official definitive site on robot exclusion. Who’s Knocking on the Door? This article by Rhoda Schueller presents a very good explanation of why a web developer would want to utilize robot exclusion. https://rietta.com/robogen/download/ Mon, 01 Jan 0001 00:00:00 +0000 https://rietta.com/robogen/download/ Download RoboGen Software This product is legacy software that is no longer maintained, nor supported by Rietta Inc. This page is preserved for historical purposes. See the listing of Rietta’s legacy software for a complete list of this software. Because RoboGen is discontinued, it is no longer available for purchase or support. If you would like to use the software, please use this complimentary license key. To activate your copy, enter the following information: Name: Complimentary RoboGen License Company: Rietta Inc. https://rietta.com/robogen/purchase/ Mon, 01 Jan 0001 00:00:00 +0000 https://rietta.com/robogen/purchase/ Free License for RoboGen 1.52 This product is legacy software that is no longer maintained, nor supported by Rietta Inc. This page is preserved for historical purposes. See the listing of Rietta’s legacy software for a complete list of this software. You are only minutes away from being able to easily control how search engines interact with your web sites. Buy your own copy of RoboGen today and begin editing your ROBOT. https://rietta.com/robogen/robots_txt/ Mon, 01 Jan 0001 00:00:00 +0000 https://rietta.com/robogen/robots_txt/ Robots.txt Tutorial This product is legacy software that is no longer maintained, nor supported by Rietta Inc. This page is preserved for historical purposes. See the listing of Rietta’s legacy software for a complete list of this software. Search engines use spider programs, also known as robots or crawlers, to create the indexes for their search databases. These crawlers follow links to find new and updated pages for the search engine.