Difference between revisions of "Retain System Overview"

Latest revision as of 18:58, 19 July 2016

Retain is made up of several parts and during setup, you are connecting all the parts together.

[edit] Major Retain Functions

When a message is dredged from the Email System the message is downloaded through the Worker to the Retain Server where the message header body parts are stored in the database, the message body, and any attachment body parts are stored on disk in the archive, the message body parts are indexed and the Reporting and Monitoring server updates statistics about the job.

[edit] How Retain Functions Are Connected

The key to a successful Retain install is getting all the connections correct.

• You connect Retain to the web server through tomcat during install.
• You connect Retain to the database during configuration.
• You connect Retain to the archive during configuration.
• You connect Retain and the Email system together through the Module.
• You connect RetainServer and RetainWorker together through the worker bootstrap.
• You connect RetainServer and Reporting & Monitoring through the Reporting & Monitoring bootstrap.

There are a number of interdependencies that need to be connected together for Retain to run smoothly:

Everything needs an Operating System (OS) to function and Retain will need a web server and a database. Since Retain is written in Java it will need Java installed and tomcat a Java web server installed on top of the regular web server.

RetainServer, RetainWorker, Reporting and Monitoring and the Indexer all sit on top of tomcat. So if you need to restart Retain it is simply a matter of restarting tomcat. In rare cases, you may need to restart tomcat, the web server, and the database. In extremely rare cases, the OS will need to be rebooted.

[edit] Retain Archiving Process Overview

When designing a Retain system it is important to know where the data goes.

1. When a job is running the Worker queries the email system for messages based on the profile and job options
2. The email system serves up a message for the Worker to process
3. The Worker opens the message, separates the email body parts and creates hashes for the body and attachments
4. The Worker sends the message body parts to the Retain Server
   1. The header data (From, Recipient, Date, &etc.) are placed in the database
   2. The body and attachments, if new, are stored on disk in the archive storage area
   3. The message body parts are indexed
   4. The actions taken by the Worker and the Server are logged in the appropriate log files

• See KB article, Retain Archive Job Flow for a more in-depth explanation of the archive job data flow.
• See KB article, Where Data Is Stored in Retain, to better understand how and where data gets stored, what needs to be backed up, and what to troubleshoot based on that.

[edit] Designing A Retain System

Retain is an archiving system and an archive is only useful if the data is kept safe. So when designing a Retain system one of the important factors to consider is data safety. We used to have many customers that installed Retain and used only a single volume for everything. Then they would call us when the disk filled up and everything crashed and data was lost. By following principles of data safety, we can minimize the effects of any volume filling up.

• The data in Retain needs to be on a different SAN from the email server, so if there is a failure on one system the other will still exist.
• The OS should be on a different volume from the data, so a bad OS update doesn't destroy data.
• The Database, Archive, Index and Logs should be on separate volumes. If any one of these volumes fills up it will not take down the system completely. This also makes it easier to monitor the system with tools like Solarwinds or Nagios. If we have the ASConfig.cfg, database, and archives we can reconstitute the system, we can reindex which only takes time and the logs are not the most important thing. In Retain 4, disk space checking has been enabled to place Retain in maintenance mode to safe the system before the drive(s) fill completely. Hint: Some sysadmins will create a multi-gigabyte file, so the disk will become full before it really becomes full, that they can delete to give them time to source additional disk space.
• We even created Reload for Retain[1] to make backups fast and easy to increase data safety, because 2 is 1 and 1 is none when it comes to backups.

See KB Article, Retain Planning & Design Best Practices

[edit] A Simple Retain System

This is appropriate for a test system or for a small organization of less than 500 active users. The simplest Retain system is made up of two servers:

• The Email server
• The Retain Server with all services on-board.

For best performance with indexing, we recommend 8 CPU cores. That seems to be the sweet spot with Retain 4.0.2.

MySQL is recommended in this case as it requires the fewest resources.

The different volumes will need 20GB free or Retain will send warnings, at 10GB free Retain will go into maintenance mode to protect the system from out of disk space errors:

• OS
• Archive
• Database
• Index
• Logs

[edit] A Distributed Retain System

A more complex Retain system can be made up of:

• Email cluster (multiple email servers. This could be individual post offices geographically distributed or a centralized Exchange DAG)
• Retain Server (There can be only one)
• Database cluster (multiple database servers)
• Retain Worker cluster (multiple workers on one or more servers)
• High Availablity Indexer cluster (multiple indexing servers, at least three)
• Reporting & Monitoring server (R&M on a separate server)

Return to Retain Training