Difference between revisions of "Retain Exchange"

Revision as of 18:29, 1 April 2015

Exchange Module On-Premise Basics

There are two major sections to setting up Retain to connect to an on-premise Exchange server.

One is the Impersonation User and the other is the Exchange Forest.

Impersonation User

In the Exchange Module the Impersonation User is formally called the Global Catalog User.

This user needs to exist in Exchange and will have a mailbox even though it won't be used. This user also needs to ApplicationImpersonation rights. Those rights can be added to a User in the 2013 Exchange Admin Console (EAC).

In the EAC under Permissions/admin roles, one of the choices should be Application Impersonation. Click on the pencil to edit and add the Impersonation user to the Members list.

If the Application Impersonation role is not already available in Permissions/Admin roles, you can create it by clicking on the plus sign. Give it a name and description and under Roles click the plus sign. Find the Display name ApplicationImpersonation and click Add ->, then ok. Add the user as a member and click save.

Appendix G of the Admin Guide describes how to set it up in Exchange 2010 & 2007.

Exchange Forest

The tab may say Exchange Forest but we are actually talking about the Active Directory Domain Forest. You will find the Domain on the Exchange Server in Active Directory Domains and Trusts or Active Directory Users and Computers.

Active Directory Directory Services is Microsoft's implementation of a directory service, it is a way to organize users, computers and other assets in an organization.

That domain at the top of the stack is the Global Catalog Host. You may have other domains and organizational units under that, but that is the one you would most want to use. At the top you have the domain, that may have multiple forests. At the other end you have organizational units made up of users and computers.

This is can be a DNS domain name or IP address. The port is 3268 for Plain Text, and 3269 for SSL connections.

The final thing to set is the Search Base. This is formatted in LDAP so it needs more information. There are 4 major components to an LDAP query:

• DC=Domain Component
• OU=Organizational Unit
• CN=Common Name
• DN=Distinguishing Name

You should become familiar with the most common errors that relate to Exchange systems.

Hands On

What does it look like if the Retain User does not have Application Impersonation Rights.

*Create your own Impersonation User account and use it as part of your Retain Exchange Module.
*Open the Exchange Admin Console.
*Create a new user for your Retain system
*Attempt to run a job. Note the error.
*Now add Application Impersonation rights.

Hands On

What does it look like if Basic Authentication is not enabled.

*Go to IIS Manager on the Exchange system and disable Basic Authentication.
*Attempt to run a job. Note the error.
*Basic Authentication Check
*Now re-enable Basic Authentication.

Hands On

What does it look like if the Search Base is incorrect. You would expect that pointing the search base of the LDAP query at the Users contain would be better as it would not have to search the entire AD forest.

*Go to Module Configuration/Exchange/Exchange Forest.
*Add CN=Users to the Search Base.
*Attempt to run a job. Note the error.
*Remove CN=Users from the Search Base.

KBs

Autodiscover: How Retain Connects to Your Exchange Mailboxes

Exchange Module Office365 Basics

O365_Setup

O365 is a little different from On-Premise as it needs two users:

• A user with Global Administrator rights for getting the user and group list via the sync365 script
• A normal user with Application Impersonation rights to collect the messages via Retain which we will refer to as the Impersonation User

Note: These accounts are based in O365 only. This is important if you are running a hybrid with On-Premise Active Directory and O365.

Administrator User

You may want to have a separate administrator user just for Retain to use for the sync365.ps1 powerscript.

• This requires Global Administrator rights.

Microsoft has a technical article on how to create an additional admin user. [1]

Impersonation User

This is just like the normal Retain Impersonation User and is setup up as an ordinary default mailbox user in the Exchange Admin Center.

Create a Normal Exchange user and make sure to give it Application Impersonation rights.

• Log into the Office 365 admin center as the Administrator user.

Creating a User for Retain

• Select Users/Active Users
• Click the Plus sign to create a new user mailbox.
  • Fill in the appropriate information.
  • Give it a simple name like RetainWorker, and have it match the logon name.
• Press Save.

Adding Application Impersonation rights

Application Impersonation is not a default right for users so you will need to set up the permissions for that and assign them to a user. Microsoft has a technical article on how to configure the system [2]

• In the Office 365 Admin Center choose Admin (near the bottom of the left navbar or from the dropdown menu at the top right) select Exchange to go to the Exchange admin center.
• Select Permissions from the left navbar, which should take you to admin roles by default
• Click the Plus sign to create a new admin role.
• Give it a name such as "Application Impersonation" or "Message Archive" and a description such as "Used by Retain to access messages".
• Under Roles click the Plus sign to add a role.
  • Select the Display Name ApplicationImpersonation, click add->, then ok.
• Under Members click the Plus sign to add a user.
  • Select the retain user, click add->, then ok.
• Press save.

sync365.ps1

This script is found under tools/exchange/Office365 and needs to be modified for each system.

It needs 3 things:

• The User Principal Name (UPN) of a user with Administrator rights.
• The Password of that user.
• The Path path to where the resulting exchangeuser.csv and exchangegroups.csv files will go, there is no need of a finale backslash. This needs to end up in the ...\RetainServer\WEB-INF\cfg directory.
  • NOTE: In powershell the backslashes '\' need to be escaped by another backslash '\\'. For example, C:\\Temp
  • NOTE: if the path has spaces then the path needs to be quoted. For example, "C:\\Retain Program\\RetainServer\\WEB-INF\\cfg"

This requires an Administrator account user UPN and password.

Automating the Office365 sync365.ps1 script [3]

Troubleshooting

One of the more useful tools in this case is Microsoft's troubleshooting tool: Microsoft Remote Connectivity Analyzer

Use the admin user specified in the sync365.ps1 script and the Impersonation user. You are looking for at least one good connection.

How Retain Authenticates against O365

1. Retain looks up the username from the csv file to get the full user information.
2. with the given credentials it does an autodiscover call, which is an SSL secured call with the username and password provided by the user.
3. if the O365 autodiscover is successful, the user has logged into Retain successfully

How It Works

Overview of Exchange

Exchange is a cluster of roles on one or more servers that transfers messages.

The major roles are:

• DNS (Domain Name Server)
• AD DS (Active Directory Directory Services)
• CAS (Client Access Server)
• Exchange Mailbox Server

The Name Server resolves which server is doing what role. It is very important that Retain and the Exchange servers all point to the same one so the domains are all resolved correctly.

Active Directory organizes the domains, users and computers of the network so they have the proper rights. Mainly that the Retain user has ApplicationImpersonation rights.

The CAS is the front door of the network, it sends requests from the users to the proper place in the network, in the case of Retain it is access to the mailbox database.

The Exchange Mailbox Server hosts the database(s) that stores the users messages.

Overview of Retain's Access to Exchange

When Retain archives from Exchange it uses the Impersonation User aka Global Catalog User to enter Active Directory (AD) to find a user.

We enter Exchange in one of four ways:

• LDAP (Lightweight Directory Access Protocol)
• SCP (Service Connection Point)
• EWS (Exchange Web Services)
• Autodiscover

In that user it searches for the user's homeMDB (homeMailboxDataBase) and uses that to link to the homeMDBBL object in AD. From there it gets the current list of mailboxes in the mail server's database.

Retain will go to the first user in that list and using ApplicationImpersonation rights. It will get the list of folders for the first user and then will process the messages.

Exchange Limitations

Exchange has certain stated limitations [4], but even in cases where there are no stated limits, there are practical limitations you are going to reach.

“I usually recommend no more than about 2500 - 5000 messages in any of the critical path folders. The critical path folders are the Calendar, Contacts, Inbox, and Sent Item folder. Ideally, keep the Inbox, Contacts and Calendar to 1000 or less. Other folders, particularly custom folders created by the user, can handle having larger numbers of items without having a broad impact on the user experience (20,000 items in my "Cookie Recipes" folder? No problem - except when I need to find that recipe from last Christmas!).” [5]

Like many computer systems it works best when it is lightly loaded which in the case of Exchange means keeping as few messages the mailbox database as is practical and since the vast majority of messages are never viewed again after they are initially received it makes sense to get them out of the system.

The average user receives 120 emails a day, 260 days of the year or 31,200 messages a year.

Throttling policy

Retain and Exchange Server 2013 Throttling Policies http://support.gwava.com/kb/?View=entry&EntryID=2343

Large Attachments and/or Messages Cannot Be Archived http://support.gwava.com/kb/?View=entry&EntryID=2089

http://technet.microsoft.com/en-us/library/dd351264%28v=exchg.150%29.aspx

Exchange Archive Strategies

Now it is certainly possible to have Exchange do its own archiving. You can set up a journaling mailbox or hold to hang on to everything as certain laws demand. If you have to keep messages for 10 years and average user may end up have 312,000 messages consuming tens of gigabytes of storage. And that is just the average user, a power user could consume far more. Can you imagine the drag on your Exchange server trying to handle all of that for thousands of users?

You could create a policy to move messages into an archive database, but since it is still connected directly to Exchange it will still be a load on the system. However there are going to be "interactions" between the holds and policies moving things to the archive database that may not allow them to be successful.

Exchange Archiving With Retain

This is where Retain comes in. Retain offloads all the messages so that Exchange can concentrate on delivering messages rather then storing them.

The typical Retain setup does not do a true archive. The customer sets up a Retain server points it at the email server and has it do a dredge every night.

For example, a message can come to a user, the user can read it, move it to the trash, and delete the trash. Has the message been removed from disk yet? No, it has not. It is moved to the Recoverable Items area of the database, where it resides for 14 days by default before being deleted. A user can undelete the message from Recoverable Items. However, they can also purge their Recoverable Items, which would remove them from the disk, not allowing Retain to dredge the message before it is deleted forever.

Unlike GroupWise, Exchange does not have message level retention abilities. So to make sure our customers are able to make proper archives there are some additional steps that need to be taken.

Journaling Mailbox

Microsoft recommends setting up a Journaling Mailbox if all messages are to be saved. There is a very large downside with this technique. If the Journaling Mailbox becomes too large, ~100GB (with various settings maximized) though it will be smaller in default systems, Exchange becomes unable to serve the messages so Retain can archive and delete them. This may work in small systems or in limited circumstances, but for the most part it is not the recommended technique for Retain.

• Exchange Journaling Mailbox Recommendations [6]

In-Place and Litigation Holds

A more effective means of archiving messages in a large environment is to set up In-Place and Litigation Holds. These holds keep items from being removed from the Recoverable Items folder for a limited time.

• Exchange Archiving with In-Place Hold [7]

A hold will keep the message in the Recoverable Items folder until the hold is released.

A good strategy for archiving is to create a rolling in-place hold so Retain has a day or two to successfully archive the messages, since Exchange doesn't have a message level flag to specify if a message has been archived or not you want to leave a little extra time to make sure Retain has a fair chance to archive.

A better strategy is to maintain the hold for 14-90 days, which will provide plenty of time to discover archiving errors and resolving them before data is potentially lost.

If users are going into Retain for past messages it is a good idea to have Retain wait to archive say 7-14 days so the messages are in their proper folders.

Installation

Exchange Installation

Retain Outlook Web Access 2013 Plugin

Retain Outlook 2013 Plugin

Troubleshooting

Troubleshooting Exchange

Bug Watch

These are the important bugs/enhancements I'm watching:

Exchange

• Bug 7128 - Excel files throw error when opened from Retain in IE
• Bug 6612 - Forwarding to Exchange causes mojibake if characters are from beyond the ASCII set

Office 365

• Bug 6679 - Add additional duplicate user checks to avoid mailbox duplication when Exchange user GUID changes FIXED in 4.0 beta
• Bug 7671 - Add additional duplicate user checks to avoid mailbox duplication when Exchange user GUID changes Need fix in 3.5.1
• Bug 5924 - Backing up the exchangeuser and exchangegroup.csv

Field Test Files

• See fix for bug 5441

Back to Retain Module Configuration