Difference between revisions of "Troubleshooting Exchange"

Latest revision as of 15:40, 6 July 2017

We do Exchange discovery look-ups in four ways:

• LDAP
• SCP
• EWS
• Autodiscover

Troubleshooting all that can be challenging.

The most common troubleshooting that needs to be done with Exchange is getting Retain to properly connect to Exchange. If you can get the module's connection test to work than the archive jobs will most likely work.

[edit] Connect Find Autodiscover Endpoint

One of the most common errors is "Connect find autodiscover endpoint" Retain is unable to connect to the expected mailbox.

Try to ping the mail server from the Retain server. If the ping cannot resolve then there is a DNS issue and a SRV may need to be added to the DNS to resolve the connection.

Confirm that Basic Authentication is enabled for ALL CAS servers for autodiscover and EWS services with the following PowerShell cmdlets:

For EWS:

Get-WebServicesVirtualDirectory | ft server,basicauthentication

For Autodiscover:

Get-AutoDiscoverVirtualDirectory | ft server,basicauthentication 

[edit] Built-in Retain Tools

Retain does have some built-in tools for helping diagnose issues connecting to Exchange.

[edit] Exchange Test

The Exchange Test is found under Module Configuration/Exchange/Exchange Forest Test Connection button. It can also be found under the Bug Icon.

NOTE: This only tests basic LDAP authentication. 
If you want a more stringent test, including autodiscover, impersonation, login, etc, 
try this link AFTER SAVING Exchange configuration:testExchange.jsp in the Util directory of RetainServer

If this test is successful you will be logged into a primitive version of the target user's mailbox. If not, then you have to check the error messages for clues to the LDAP errors.

[edit] WorkerLog

You will find a starting point for troubleshooting LDAP issues in the worker log.

The log will return an error something like this (only horizontally formatted):

10:07:52,746 LiveEWSUserSelection - 
javax.naming.NameNotFoundException: 
 [LDAP: error code 32 - 0000208D: NameErr: DSID-031522C9, problem 2001 (NO_OBJECT), data 0, best match of:
'CN=InformationStore,
CN=EXCH01,
CN=Servers,
CN=Exchange Administrative Group (EYDIBOHF23SBDLT),
CN=Administrative Groups,
CN=Gwava Organization,
CN=Microsoft Exchange,
CN=Services,
CN=Configuration,
DC=gwava,
DC=local'
 ]; remaining name 
'CN=Mailbox Database,
CN=First Storage Group,
CN=InformationStore,
CN=EXCH01,
CN=Servers,
CN=Exchange Administrative Group (EYDIBOHF23SBDLT),
CN=Administrative Groups,
CN=Gwava Organization,
CN=Microsoft Exchange,
CN=Services,
CN=Configuration,
DC=gwava,
DC=local'

What you will want to do is see if you can use LDAP Browser or one of the other tools to reach those containers in Active Directory, it is important to do that with the ImpersonationApplication user because that is what Retain uses to traverse the tree. If logged in as an administrative user you will not be able to tell if the ImpersonationApplication user can traverse the tree to those containers. The location data is read from right to left, eg from most specific to least specific: CN=Mailbox Database is more specific then DC=gwava,DC=local.

[edit] RetainWorker/diag/ex

If you log into the Retain Worker there is another tool. It looks much the same as the Exchange Test but also requires the Retain server IP address or name.

This too will log you into a primitive version of the target user's mailbox.

[edit] Active Directory Tools

There are three major tools for accessing Active Directory(AD) on the domain controller server. These can be accessed from the Tools menu of Server Manager or from the Start Menu.

• Active Directory Users and Computers
• Active Directory Domains and Trusts
• Active Directory Sites and Services

[edit] Active Directory Users and Computers

This is where users are organized and managed even if they are created in the Exchange Admin Center. Here is where organizational units are created and users placed in them. This is also where you want to go to make sure that the Global Catalog User in Retain has the proper rights and privileges.

[edit] Active Directory Domains and Trusts

If an organization has multiple domains this is where those are managed. [1]

[edit] Active Directory Sites and Services

When Retain is searching for the user list for a job, as shown in the worker log, it will look in the Services container, which can be accessed in this tool.

One thing to note the Services node is not visible by default:

• To make the services node visible,
• Select the domain (which should be the top container in the left side bar)
• Go to the View menu
• Click "Show Services Node"

Now you will be able to follow the trail that Retain took looking for the user list when the job started. This is described under Worker Log below.

[edit] MS Remote Connectivity Tester

Microsoft has a useful tool called the Remote Connectivity Tester [2] This online tool that can be helpful to narrow down issues. It is something the customer can use to narrow down the troubleshooting search. It will test the ways it expects to be able to access a system. It will give a very complete report and you will have to go digging to find the particular error. Many of which will be the tool trying to access the system using standard but unavailable entrances.

[edit] LDAP

LDAP (Lightweight Directory Access Protocol) is a platform independent way to access directory services. [3]

LDAP Basics

[edit] LDAP Browser

[4]

This is just a browser and allows you to read and traverse your Active Directory forest.

The thing is to log in using the credentials you gave Retain, if you can do that it will work for an archive job. But in real life it will be more difficult than that.

1. Log in as the Administrator user to find the actual credentials and path of the ApplicationImpersonation account.
2. Then go back and figure out how to log in to AD with them and see all of the tree.
3. Then make sure Retain is set with those properties and use the EWSEditor Log Viewer Tool to find how it works or fails.

[edit] Impersonation Account troubleshooting

If a customer is having issues with the Impersonation Account User this is a useful tool to troubleshoot the problem. [5]

• Log into the Active Directory forest as an administrator so you have full rights.
• Drill down to the user retain uses for the ImpersonationAccount. The things you are looking for are:
  • userPrincipalName: that is the name you should be using to log into Exchange. It should look like their email address.
  • memberOf: the user must be a member of CN=Application Impersonation
  • userAccountControl: should be NormalAccount (it cannot be an admin account or it will not work due to Microsoft security features) and NoPasswordExpiration (so you never have to worry about jobs failing because the worker can't log in anymore)
  • homeMDB: this shows that the retain user has a mailbox account on the server which is required.

• In the Exchange Module/Impersonation tab. The Global Catalog User needs to be the userPrincipalName of the ImpersonationAccount user and the Global Catalog Password is the matching password.

• Now to test the Impersonation account user, create a new session in LDAPBrowser. File/New/New Profile and login using the Impersonation User credentials (but in this case it will be the distinguishedName). The user should be able to navigate to the Users container and any OU containers users reside in. If the Impersonation User can access all these containers then Retain will be able to archive all of them as well.

[edit] User AD Data export

If there is a particular user that has issues you can log into LDAPBrowser and export their data.

1. Go to that user and File/Export Data.
2. A wizard will appear and you will want to export as a .csv file.
3. Save to a place you can find it again later.

[edit] homeMDBBL export

Retain will do an LDAP lookup in Active Directory on the user object and read its properties. It is looking for the homeMDB property and its value. If autodiscover finds the autodiscover server but cannot find the user's mailbox, login to Active Directory as the impersonation account using an LDAP browser and see if you can read the user's properties, especially the homeMDB value.

This is also helpful when you want to find out what mailbox database a particular user is a part of (although that is also shown in the user object properties in the Exchange Admin console), find the user object in Active Directory that is known to be part of that database and double-click on the homeMDB property. That will jump you to a different place in the directory tree which is normally hidden. Something along the lines of:

CN=Mailbox Database 0766879867,
CN=Databases,
CN=Exchange Administrative Group (FYDIBOHF23SPDLT),
CN=Administrative Groups,
CN=sol,
CN=Microsoft Exchange,
CN=Services,
CN=Configuration,
DC=ad,
DC=sol,
DC=net

There will be a list of homeMDBBL showing the users in that database. From here you can do a data export that will provide a list of all users in that database.

[edit] SCP

Service Connection Point

[6]

[edit] EWS

Exchange Web Services is how clients systems can connect to an Exchange server. [7]

[edit] Exchange Management Shell

You can get a quick overview of the customers Exchange system by asking it. In the Exchange Management Shell, not the regular powershell, on the Exchange server run the following command:

Get-WebServicesVirtualDirectory | fl >C:\ews.txt

The results should look something like this:

RunspaceId                      : 33ac48e8-9ff0-461f-b604-6ea6ef8a3bf4
CertificateAuthentication       : 
InternalNLBBypassUrl            : 
GzipLevel                       : Low
MRSProxyEnabled                 : False
Name                            : EWS (Default Web Site)
InternalAuthenticationMethods   : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth}
ExternalAuthenticationMethods   : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth}
LiveIdNegotiateAuthentication   : 
WSSecurityAuthentication        : True
LiveIdBasicAuthentication       : False
BasicAuthentication             : True
DigestAuthentication            : False
WindowsAuthentication           : True
OAuthAuthentication             : True
AdfsAuthentication              : False
MetabasePath                    : IIS://EXMS.support.LOCAL/W3SVC/1/ROOT/EWS
Path                            : C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\EWS
ExtendedProtectionTokenChecking : None
ExtendedProtectionFlags         : {}
ExtendedProtectionSPNList       : {}
AdminDisplayVersion             : Version 15.0 (Build 775.38)
Server                          : EXMS
InternalUrl                     : https://mail.support.com/EWS/Exchange.asmx
ExternalUrl                     : https://mail.support.com/ews/exchange.asmx
AdminDisplayName                : 
ExchangeVersion                 : 0.10 (14.0.100.0)
DistinguishedName               : CN=EWS (Default Web Site),CN=HTTP,CN=Protocols,CN=EXMS,CN=Servers,
                                  CN=Exchange Administrative Group (FYDIBOHF23SPDLT),
                                  CN=Administrative Groups,CN=Exchange,CN=Microsoft 
                                  Exchange,CN=Services,CN=Configuration,DC=support,DC=LOCAL
Identity                        : EXMS\EWS (Default Web Site)
Guid                            : 62be879a-5a5c-495a-8cca-481bfd1c40c2
ObjectCategory                  : support.LOCAL/Configuration/Schema/ms-Exch-Web-Services-Virtual-Directory
ObjectClass                     : {top, msExchVirtualDirectory, msExchWebServicesVirtualDirectory}
WhenChanged                     : 3/20/2014 1:49:19 PM
WhenCreated                     : 2/21/2014 11:22:32 AM
WhenChangedUTC                  : 3/20/2014 8:49:19 PM
WhenCreatedUTC                  : 2/21/2014 7:22:32 PM
OrganizationId                  : 
OriginatingServer               : LA-DC1.support.LOCAL
IsValid                         : True
ObjectState                     : Changed

RunspaceId                      : 33ac48e8-9ff0-461f-b604-6ea6ef8a3bf4
CertificateAuthentication       : 
InternalNLBBypassUrl            : 
GzipLevel                       : Low
MRSProxyEnabled                 : False
Name                            : EWS (Default Web Site)
InternalAuthenticationMethods   : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
ExternalAuthenticationMethods   : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
LiveIdNegotiateAuthentication   : 
WSSecurityAuthentication        : True
LiveIdBasicAuthentication       : False
BasicAuthentication             : False
DigestAuthentication            : False
WindowsAuthentication           : True
OAuthAuthentication             : True
AdfsAuthentication              : False
MetabasePath                    : IIS://EXMS2.support.LOCAL/W3SVC/1/ROOT/EWS
Path                            : C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\EWS
ExtendedProtectionTokenChecking : None
ExtendedProtectionFlags         : {}
ExtendedProtectionSPNList       : {}
AdminDisplayVersion             : Version 15.0 (Build 775.38)
Server                          : EXMS2
InternalUrl                     : https://mail.support.com/EWS/Exchange.asmx
ExternalUrl                     : https://mail.support.com/EWS/Exchange.asmx
AdminDisplayName                : 
ExchangeVersion                 : 0.10 (14.0.100.0)
DistinguishedName               : CN=EWS (Default Web Site),CN=HTTP,CN=Protocols,CN=EXMS2,CN=Servers,
                                  CN=Exchange Administrative Group (FYDIBOHF23SPDLT),
                                  CN=Administrative Groups,CN=Exchange,CN=Microsoft 
                                  Exchange,CN=Services,CN=Configuration,DC=support,DC=LOCAL
Identity                        : EXMS2\EWS (Default Web Site)
Guid                            : 35cf96b1-29b4-442e-81c2-93298a8526af
ObjectCategory                  : support.LOCAL/Configuration/Schema/ms-Exch-Web-Services-Virtual-Directory
ObjectClass                     : {top, msExchVirtualDirectory, msExchWebServicesVirtualDirectory}
WhenChanged                     : 3/20/2014 1:49:37 PM
WhenCreated                     : 3/13/2014 10:30:45 AM
WhenChangedUTC                  : 3/20/2014 8:49:37 PM
WhenCreatedUTC                  : 3/13/2014 5:30:45 PM
OrganizationId                  : 
OriginatingServer               : LA-DC1.support.LOCAL
IsValid                         : True
ObjectState                     : Changed
 

This is a quick report of the systems settings. The important thing to check is that BasicAuthentication is set to True.

[edit] EWSEditor

Sometimes you may have to dig deeper into a system to determine what is wrong.

EWSEditor [8] is a powerful and more dangerous tool that LDAP Browser since it can change entries.

1. Log in as the Impersonation user to the Impersonation Mailbox. This is a very picky tool so you will have to do it right, see LDAP Browser. [9]
2. Use the Autodiscover Viewer Tool to attempt to access other mailboxes on the system and look in the EWSEditor Log tool to see what steps it is going through to reach the user.

[edit] Login into EWSEditor with Impersonation Rights

To test the Impersonation User you want to log into EWSEditor with the impersonation user mailbox with the impersonation user credentials.

1. Specify the Service URL: to be something along the lines of https://10.1.4.200/EWS/Exchange.asmx and set the Exchange version to your own.
2. Check Use the following credentials instead of the default Windows credentials
3. User Name is the mailNickName or sAMAccountName and fill in the password and domain.
4. Check Use impersonation to log on to another mailbox using the credentials specified on the credentials tab by identifying the mailbox Id below.
   • Id Type: set to PrincipalName
   • Id: set to the impersonation user's UPN e.g., Retain@ad.sol.net

If that fails then the customer will have to call Microsoft since there is something set incorrectly in AD that is not allowing Impersonation rights to function correctly.

[edit] Autodiscover Tool

EWSEditor has an autodiscover tool under Tools/Autodiscover Viewer.

1. Uncheck Default Windows Credentials and enter the Impersonation User Credentials
2. Enter the SMTP address of the mailbox you are attempting to access and the Exchange version you are running and press run.
   • If you are successful you will see data returned.
   • Otherwise an error will be returned. For more detail you can go back to the EWSEditor and select Tools/EWSEditor Log Viewer and see the steps it attempted in detail.

[edit] Autodiscover

Autodiscover is a way Microsoft provides that allows for software objects to be configured. [10]

We expect an address like https://autodiscover.mail.gwava.net/EWS/exchange.asmx or https://autodiscover.mail.gwava.net/autodiscover/autodiscover.xml

[edit] IIS

Autodiscover is part of IIS, so when changes are made IIS will have to be restarted.

To restart IIS immediately run PowerShell as Administrator and 
run the command "IISRESET" and it will restart IIS immediately.

[edit] Ports

Autodiscover ports in Firewall [11]

• External Interface is Protocol- Https, TCP/UDP-TCP,Port-443
• Internal Interface Protocol- HTTPS, TCP/UDP-TCP, Port-4443

[edit] Endpoint Error during migration

If the customer has multiple email domains but only one one AD domain you will see this error.

Autodiscover needs to be added to your DNS .srv file. Microsoft has a kb on that subject [12]

In the DNS manager create a new Forward Lookup Zone

The general format is:

• Service: _autodiscover
• Protocol: _tcp
• Port Number: 443
• Host: mail.gwava.com

[edit] Exchange Autodiscover kbs

• Exchange Autodiscover [13]

• Exchange Autodiscover (Part 2) [14]

• White Paper: Exchange 2007 Autodiscover Service [15]

• A new feature is available that enables Outlook 2007 to use DNS Service Location (SRV) records to locate the Exchange Autodiscover service [16]

[edit] Other Items

Some basic KBs:

• How to Create a SOAP Trace for Exchange Jobs [17]
• Location of Exchange Logs [18]

For troubleshooting IE Microsoft has VMs for different versions of IE. [19]

[edit] HealthMailbox

These are system monitoring mailboxes and almost invisible. Exclude them from jobs. If left to their own devices they will fill to the quota or more and then a new one will be created. which will have to be excluded manually.

Ideally they would be have a retention policy that deletes content after a day but it seems like they just fill up. But that doesn't seem to work [20]

However you may be able to create a script to empty them on a regular basis. [21]

[edit] Dynamic Distribution Lists

Are basically filters that create user lists when messages are sent, so Retain cannot key off of them in the exchangegroups.csv file.

[edit] Exchange Logs

You can find the logs on the Exchange server. And there are a lot of log files. However the one you generally want to look at is in: D:\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

EWS is one of the main routes Retain uses to connect with Exchange. There are new logs created every hour, so you will need to know when the error happens in Retain. The Exchange logs are timestamped in GMT (Z or Zulu) so you will need to take that into account. Retain logs are timestamped to the local server time.

When looking for something in the Exchange log you can search for the impersonation username. In this case the impersonation username is "retain".

2015-05-27T17:55:38.733Z,85599143-9c67-4fe4-8785-fe03858e2c37,15,0,847,30,,Ews,exmail.ad.gwavasupport.com,/EWS/Exchange.asmx,,Basic,True,support\retain,,Sid~S-1-5-21-132121869-3271972796-1041755442-1154,JAX-WS RI 2.1.7-hudson-48-,10.1.4.40,EXMAIL,200,200,,POST,Proxy,exmail.ad.gwavasupport.com,15.00.0847.000,IntraForest,WindowsIdentity,Database~21ff040a-4051-4678-9549-b3cea0ed96cc~~06/26/2015 17:55:38,,,6642,7625,1,,1,0,,0,,0,,0,0,124.8752,0,1,0,0,114,0,4,1,122,0,120,2,2,3,123,,,CorrelationID=<empty>;BeginRequest=2015-05-27T17:55:38.608Z;ProxyState-Run=None;BeginGetRequestStream=2015-05-27T17:55:38.608Z;OnRequestStreamReady=2015-05-27T17:55:38.608Z;BeginGetResponse=2015-05-27T17:55:38.608Z;OnResponseReady=2015-05-27T17:55:38.717Z;EndGetResponse=2015-05-27T17:55:38.717Z;ProxyState-Complete=ProxyResponseData;S:ServiceCommonMetadata.Cookie=1cd8e6f6db17494db7cd9386b5ddc5be,

[edit] IIS Configuration

There are a few files that control the configuration of IIS 7 and 8. [22] There are several pages of interest in there.

[edit] Stuck Queues

Sometimes messages get stuck in the Exchange queues. This can cause the Exchange server to run slowly as it uses lots of CPU cycles.

To get a quick overview of the queues in the EMS

Get-Queue

You can also use the Exchange Toolbox and the Queue Viewer found with it.

Once, a test system was bogged down really bad, CPU was >90% and it was difficult to reach in any way. The Unreachable queue had over 180,000 messages. Trying to remove the messages in Queue Viewer failed. Restarting the server failed, but restarting the Edge Transport, Mailbox Transport Delivery and/or SMTP service allowed Exchange to clear the messages. CPU usage quickly dropped to <30%.