Difference between revisions of "Retain Blackberry"
(→Installation and Setup) |
|||
(12 intermediate revisions by 4 users not shown) | |||
Line 37: | Line 37: | ||
direct file access to the logs. This access can be accomplished through a shared folder or location, but it must be available to the Retain worker. Appropriate access through firewalls or | direct file access to the logs. This access can be accomplished through a shared folder or location, but it must be available to the Retain worker. Appropriate access through firewalls or | ||
network security must be made available if the worker is not installed on the BES system. | network security must be made available if the worker is not installed on the BES system. | ||
+ | |||
+ | '''BES Server''' - This portion is designed to configure communication with the BlackBerry Server for the Address Book Sync. | ||
+ | Server Name - The server name can be anything you would like. It is good practice to name this something unique to the BES Server its configured for | ||
+ | Logfile Path - The path to your BlackBerry Enterprise Server log files. For this portion the logfile path is not verified | ||
+ | SOAP Host - This is the IP address of Host name of the BlackBerry Enterprise Server | ||
+ | SOAP Port - BES 5: 443 | BES 10: 38443 | BES 12: 18084 | ||
+ | SOAP User - An Administrative user on the BlackBerry Administration Service | ||
+ | SOAP Password - Password for configured user | ||
'''SMTP Connector''' - In order to allow the forwarding of BES items out of the Retain system, the Blackberry Module needs to be configured with an SMTP connector. This connector will | '''SMTP Connector''' - In order to allow the forwarding of BES items out of the Retain system, the Blackberry Module needs to be configured with an SMTP connector. This connector will | ||
Line 211: | Line 219: | ||
monitoring tool to report on exactly how much mail and mailboxes have been archived, along with any errors | monitoring tool to report on exactly how much mail and mailboxes have been archived, along with any errors | ||
encountered. | encountered. | ||
+ | |||
+ | ==How It Works== | ||
+ | |||
+ | ==BBM Protected Overview (Plus Many Other Topics) Training Video== | ||
+ | Below is a video recording from July 31, 2015 given by our product manager, Colin Bretagne. It requires a video player that can play a .webm file. You can download one of those players here: [http://www.videolan.org/vlc/ VLC Video Player] | ||
+ | |||
+ | Click on this link for the training video: [http://training.gwava.com/videos/Retain4-Mobile-Social-BB-etc_Training_by_Colin.webm Video of training given my Colin Bretagne July 31, 2015] | ||
==Installation and Setup== | ==Installation and Setup== | ||
Line 216: | Line 231: | ||
In order to archive for BlackBerry certain steps need to be in place | In order to archive for BlackBerry certain steps need to be in place | ||
− | + | - The users device needs to be in Regulated Mode | |
− | + | If the user device is not set to regulated mode the information will not be available for archive | |
− | + | - The BlackBerry Enterprise Server needs to provide log files for Retain to Archive from. | |
− | + | Retain will Dredge the log file created by the BlackBerry Enterprise server in order to gather the information it needs for the Archive. | |
− | + | - The BlackBerry Module needs to be configured on the Retain Server | |
− | + | Configuring the Blackberry module is done in a few short steps. | |
− | + | - Enable the job. If the enable job button is not checked no jobs will be sent to the Retain Worker | |
− | + | - Blackberry Enterprise server configuration. You can name the server anything you would like. The | |
− | + | log file path is critical to the success of a job. The location of the Blackberry Enterprise log files | |
− | + | is entered. The location must be either an NFS share or a Samba Share. | |
+ | |||
+ | - A Schedule, Profile, Worker and Job need to be created on the Retain Server. | ||
+ | - The schedule determines how often the job will archive new data | ||
+ | - The profile allows you to set what type of messages you will archive, what scope/range of messages will be included, etc | ||
+ | - The worker is responsible for the communication between the Retain Server and the Blackberry Enterprise Server. | ||
+ | - The Job is where all the components come together. You will chose the schedule, profile and worker that you want to use. | ||
+ | You can also select what mailboxes you want to archive. You can archive an entire system or specific users | ||
+ | |||
+ | ==Archive Display and Interaction== | ||
+ | |||
+ | Archived information is displayed within the users 'store' according to category. General information about the message/phone call is displayed on the main screen. Such as the From, Recipient, subject and date. | ||
+ | |||
+ | Opening the message will provide specific details about the message. You will be presented with a Message tab and a Properties tab. | ||
+ | -The message tab will show you who the message is To and From, the subject and will show you the contents of the body of the message. | ||
+ | -The properties tab gives specific information | ||
+ | - Item Type - Type of communication (BBSMS, BBMMS, etc) | ||
+ | - Box Type - Tells you if the message/phone call was sent or received | ||
+ | - Owner UID - Users Blackberry PIN | ||
+ | - Read - if the message was opened or not | ||
+ | - Confidential | ||
+ | - Indexing - Tells you if the message has been indexed for searching | ||
+ | - Created - Created date | ||
+ | - Delivered - Date Delivered | ||
+ | - Stored - Date stored in Retain | ||
+ | - Expiration Date - Date set for deletion if deletion management has been setup in Retain | ||
+ | - Begin Date - Shows time lapse of message or phone call | ||
+ | - End Date - Shows time lapse of message or phone call | ||
+ | - Path - Give the directory path for the message/phone call | ||
+ | Additional Properties | ||
+ | - postoffice - Post office that holds/held the message | ||
+ | - email System | ||
+ | - domain | ||
+ | - internet | ||
+ | - abSystem | ||
+ | Audit | ||
+ | - Information under this section will show an audit trail. Such as when the message was archived, the IP address that archived it. | ||
+ | |||
+ | |||
+ | ==Blackberry logging== | ||
+ | BES Log Reading Tips | ||
+ | |||
+ | The logs for BES data will be in the RetainServer log | ||
+ | ===Dev overview=== | ||
+ | The Retain AddressBook sync was written a few years ago using SOAP to connect to a BES10 server. It has been tested with a BES5 and BES12 system since then using the same code with slight modifications. This code was written using the BES10wsdl.wsdl and BES10Utilwsdl.wsdl that I believe came from BlackBerry. | ||
+ | |||
+ | The administrator configures the BES credentials in Retain (host, port, username and password). The Retain AddressBook code will use this information to make a SOAP connect from the RetainServer to the BES server. The url being used would end up being something like: | ||
+ | |||
+ | https://bes10.qa.gwava.com:38443/enterprise/admin/ws | ||
+ | |||
+ | The code starts by making a request for GetSystemInfoRequest information. The getSystemInfo() call requesting the metadata (REQUEST_METADATA) looking for the "BWS Version" is for logging purposes but also to understand how to interpret the values being retrieved from a BES5, BES10 and BES12 system. | ||
+ | |||
+ | The code then sets up a request for the GetUsersRequest list. The GetUsersSortBy is set up with the setDISPLAYNAME(true) and the setValue("DISPLAY_NAME"). The getUsers() request is called with the setSearchCriteria and the SetSortBy being set. The setSortAscending(true), setPageSize(100) and setMetadata(REQUEST_METADATA) are configured. At this point the getUsers() response generates a user list in memory, wanting to fill in some extra information. | ||
+ | |||
+ | In order to get more information, the code then sets up a request (url admin/ws) for the GetUsersDetailRequest details. The setMetadata(REQUEST_METADATA), getLoadUserEmailProfiles(true), getLoadAccounts(true) and setLoadDevices(true) are configured. The code adds in each user returned from the user response in order to request the individual's details (.add(user)). At this point the getUsersDetail() response generates a list of details. | ||
+ | |||
+ | First - the DisplayName, FirstName, and LastName are pulled out. | ||
+ | |||
+ | Second - a list of email addresses is retrieved using the getDirectUserEmailProfiles() call, tagging the first email found using the getEmailAddress() as the primary email address and the corresponding userId found using the getUsername() or the getAccountName(). | ||
+ | |||
+ | Third - a list of accounts is retrieved using the getAccounts() call, tagging (if needed) the first account found using the getEmailAddress() as the primary email address, the corresponding userId found using the getMailboxDN() and the corresponding userConfigId using the getUid() value (everything after the last hyphen "-" in it for future usage). | ||
+ | |||
+ | Finally - a list of devices is retrieved using the getDevices() call, storing all the BB Pins found using the getBlackBerryPIN() and the PhoneNumbers using the getPhneNumber(). | ||
+ | |||
+ | If no primary email address and no device information is found, the user is ignored. | ||
+ | |||
+ | |||
+ | ===BES Address Book Sync=== | ||
+ | |||
+ | For the address book you’ll want to pull out the lines with [AddressBookSyncThread], just to make it easier to handle. | ||
+ | |||
+ | The address book sync will start with | ||
+ | 01:02:05, 663[AddressBookSyncThread] [INFO ] SyncABookJob: Run address book sync for blackberry | ||
+ | 01:02:05, 663[AddressBookSyncThread] [INFO ] SyncABookJob: addressBookSync: com.gwava.address.sync.BlackBerryAddressBookSync | ||
+ | 01:02:05, 663[AddressBookSyncThread] [INFO ] BlackBerryAddressBookSync: Starting BlackBerry Address Book sync ... | ||
+ | 01:02:05, 788[AddressBookSyncThread] [DEBUG] BlackBerryAddressBookSync: Getting SOAP configuration for bizcorp.com | ||
+ | 01:02:05, 788[AddressBookSyncThread] [DEBUG] BESAddressBook: connectToServer - connecting to <server named in module> at <IPaddress or hostname>:<port>(usually 80 or 443) | ||
+ | |||
+ | Getting User List, emails and PIN or Phone Numbers | ||
+ | When the BES AddressBook lookup occurs you should see the following log for each user it finds: | ||
+ | |||
+ | "getUserList - collecting details for user", followed by displayName, followed by BESUid. A BES 10 user will have a 4-6 digit number, a BES 12 user will have an alphanumeric GUID | ||
+ | Something like: | ||
+ | BES 10 | ||
+ | 01:03:02, 500[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, John, 32621 | ||
+ | BES 12 | ||
+ | 07:19:57, 661[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Roe, David, 6388607a-1235-4347-85d1-652070e46d85 | ||
+ | |||
+ | Then it looks in the BES email profiles for the email address. No direct logging of this action. | ||
+ | |||
+ | Then it looks a second time for email address in the BES Accounts. No direct logging of this action. | ||
+ | |||
+ | Then it looks in the BES devices for the pin and the phone number. No direct logging of this action. | ||
+ | |||
+ | If it finds nothing in the devices section (meaning the pin and phone number) it logs: | ||
+ | |||
+ | "getUserList - ignoring user", followed by display name, followed by "phone number and PIN not available". | ||
+ | |||
+ | And no user and device will be created or updated in the device management table. | ||
+ | |||
+ | If it finds something then it logs: | ||
+ | |||
+ | "getUserList - BES 10 or 12 generated nameId", followed by the nameId | ||
+ | |||
+ | Otherwise it logs: | ||
+ | |||
+ | "getUserList - ignoring user", followed by display name, followed by "email address not available", | ||
+ | |||
+ | Then nothing should be created or updated in the device management table. | ||
+ | |||
+ | Examples: | ||
+ | If a user is found with a Display Name and email address and PIN or Phone# for a device, we’ll see: | ||
+ | 01:03:02, 500[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, John, 32621 | ||
+ | 01:03:03, 436[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - BES 10 or 12 generated nameId bizcorp.com/2125551212 | ||
+ | |||
+ | If a user has a Display Name and Email but no device, we’ll see: | ||
+ | 05:37:31, 668[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, Jane, 33234 | ||
+ | 05:37:31, 668[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - ignoring user Doe, Jane, phone number and PIN not available | ||
+ | |||
+ | If a user has a Display name but no email, we’ll see: | ||
+ | 05:07:01, 640[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Person, Person, 4447 | ||
+ | 05:07:01, 640[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - ignoring user Person, Person, email address not available | ||
+ | |||
+ | The address book sync will end with | ||
+ | 09:15:39, 536[AddressBookSyncThread] [DEBUG] BlackBerryAddressBookSync: Refreshing the device cache | ||
+ | 09:15:39, 536[AddressBookSyncThread] [DEBUG] RetainDBStorageEngine: Priming the device cache by reading in the list of devices | ||
+ | 09:15:46, 634[AddressBookSyncThread] [DEBUG] RetainDBStorageEngine: Loaded 101628 device(s) into the device cache | ||
+ | 09:15:46, 634[AddressBookSyncThread] [INFO ] BlackBerryAddressBookSync: BlackBerry Address Book sync finished | ||
+ | 09:15:46, 649[AddressBookSyncThread] [INFO ] SyncABookJob: AddressBook job completed | ||
+ | |||
+ | ===BES Message Archiving=== | ||
+ | |||
+ | You’ll want to pull out the message storage thread listed with "[REST-Default-" | ||
+ | There are three kinds of messages: BBPIN (text PIN messages), BBSMS (text SMS messages), and BBMMS (multimedia MMS messages) | ||
+ | |||
+ | An individual message will look something like this: | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Checking authorization with com.gwava.restapi.security.auth.application.OneRightApplicationAuthorizer@73701a09 for the application Retain Worker | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Application Retain Worker successfully authorized. | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Checking authorization with AuthorizerDisjunctionChain [authorizers=PrincipalTypeAuthorizer [requiredType=Token] OR PrincipalTypeAuthorizer [requiredType=Collector]] for principal RetainCollectorPrincipal [primaryUID=B7CD9A6307A4A9903E2BC3AC0013D87A, secondaryUIDs=UIDSet [uidSet=[]], aclSet=UserACLSet [aclSet=[ADMIN_MANAGEWORKERS, USER_DELETEOTHER, USER_EXPORT, ADMIN_SEECONFIDENTIAL, USER_FORWARD, ADMIN_MANAGEUSERS, ADMIN_GLOBALTAGDEF, ADMIN_AUDIT, ADMIN_DELETE, ADMIN_MANAGESERVER, ADMIN_SEARCH, ADMIN_GROUPTAGDEF, USER_AUDIT, ADMIN_PUBLISH, USER_USERTAGDEF, USER_PRINT, ADMIN_ACCESSRM, USER_READWORKER, USER_SELFCONFIDENTIAL, USER_APPLYCONFIDENTIAL, ADMIN_RESTORE, USER_ATTACHMENT, USER_DELETEPERSONAL, ADMIN_LITIGATE, USER_RESTORE]]] | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: User B7CD9A6307A4A9903E2BC3AC0013D87A successfully authorized. | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] PreauthorizedProcessor: Authorization succeeded; calling performOperation | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: InsertArbitraryMessagesProcessor.performOperation called | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: We will attempt to archive 1 messages now. | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: We also have 1 bytestreams to process. | ||
+ | 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: InsertArbitraryMessagesProcessor.performOperation exit | ||
+ | 00:01:43, 359[REST-Default-3355] [TRACE] InsertMessageConverters: NewMessageToArchiveEmailParams.convert called for DEB77E29C1C4E84B4C86597428E6E557 | ||
+ | 00:01:43, 359[REST-Default-3355] [DEBUG] FolderPathCreator: getOrCreate Folder started. | ||
+ | 00:01:43, 359[REST-Default-3355] [DEBUG] CacheProvider: Folders - Looking for element with cacheKey : 86C69E078AF2B44055FBE013F7D37CFE | ||
+ | 00:01:43, 359[REST-Default-3355] [TRACE] CacheProvider: Inserting element with cacheKey : 86C69E078AF2B44055FBE013F7D37CFE | ||
+ | 00:01:43, 359[REST-Default-3355] [DEBUG] CacheProvider: Folders - Looking for element with cacheKey : BE3B7391BADCAE2223DC45D755B690CD | ||
+ | 00:01:43, 375[REST-Default-3355] [TRACE] CacheProvider: Inserting element with cacheKey : BE3B7391BADCAE2223DC45D755B690CD | ||
+ | 00:01:43, 375[REST-Default-3355] [TRACE] FolderPathCreator: getOrCreate Folder ended, returning 3879 | ||
+ | 00:01:43, 391[REST-Default-3357] [TRACE] InsertMessageProcessors: MessageInsertionMapper: about to insert 1 messages. | ||
+ | 00:01:43, 391[REST-Default-3357] [DEBUG] SingleProcessStorageEngine: receiving document Are you still up? .txt done. hash:20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB | ||
+ | 00:01:43, 406[REST-Default-3357] [INFO ] RetainDBStorageEngine: About to commmit transaction null | ||
+ | 00:01:43, 406[REST-Default-3357] [DEBUG] RetainDBStorageEngine: [HibernateUtil.insertMessage]: , msgID: DEB77E29C1C4E84B4C86597428E6E557, uuid: 3cdad518-e3d8-467e-9dc0-eadcfabe9517, doc: com.gwava.dao.social.Document id=644326, hash=20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB | ||
+ | 00:01:43, 406[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Looking up device by phone number 2125551212 | ||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Message was present: Are you still up? , doc: com.gwava.dao.social.Document id=644326, hash=20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB | ||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: TRANSACTION: skipping double attachment | ||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Are you still up? - attachments hasChanged: false | ||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: "Are you still up? .txt" has parent: \doed\BlackBerry\BBSMS isSubmail: false has changed: false | ||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: check if indexing needed... | ||
+ | 00:01:43, 422[REST-Default-3357] [TRACE] RetainDBStorageEngine: Item 7575211 has not been added to index queue because it has been indexed. | ||
+ | 00:01:43, 453[REST-Default-3358] [TRACE] SpringAsyncUtils: UpdateResultOnSuccess: Setting DeferredResult to success value: <200 OK,com.gwava.retain.rest.v1.insertion.impl.InsertMessageResponseImpl@7677d2a5,{}> | ||
+ | |||
+ | For a message, you can find the user by the nameID from the address book sync: For example if the users Display Name is David Doe and nameID is "doed" | ||
+ | |||
+ | 05:27:28, 899[REST-Default-3453] [DEBUG] RetainDBStorageEngine: "Fw: Test1.txt" has parent: \ David Doe\BlackBerry\BBPIN isSubmail: false has changed: false | ||
+ | |||
+ | 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: "Where are you? .txt" has parent: \doed\BlackBerry\BBSMS isSubmail: false has changed: false | ||
+ | |||
+ | 06:37:18, 458[REST-Default-3469] [DEBUG] RetainDBStorageEngine: "Can these to get any cuter?!?! I swear I'm comin.txt" has parent: \doed\BlackBerry\BBMMS isSubmail: false has changed: false | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | ==Back to [ | + | =====Back to [[Retain Module Configuration]]===== |
Latest revision as of 13:39, 24 September 2018
The BlackBerry module allows the collection of BlackBerry phone data from the Enterprise server into the Retain system. BES 5.x and 10.x are supported.
To archive the BlackBerry information, the BlackBerry Enterprise Server must be configured to log phone calls, PIN, BBM and SMS data. The server log files are what Retain uses to collect the information about phone activity, and as such no software or modification needs to be made to any system phones. If the BlackBerry server has not been configured to log the data, Retain will be unable to archive the information.
For BES 5.x, instructions to modify the logging in the BlackBerry Enterprise Server can be found in the BlackBerry Enterprise Server help file. (Found under Start | Programs | BlackBerry Enterprise Server | Help | BlackBerry Manager Help)
For BES 10.x, users must be set on EMM – Regulated mode and have the logging set to ‘yes’ for all types desired. Balanced mode users will not have logs created for them in the BES system and as a result will not be archived.
Installing the BlackBerry Module is incredibly simple. Essentially, the module needs to be enabled and configured with the BES log file location. Afterwards, the data is treated the same as any other data entering the Retain archive. Data is archived in the database under the user/phone it originated from and can be searched and viewed through the Search Messages interface.
Contents |
[edit] BlackBerry Module Configuration
- Open an Internet Browser and go to your Retain Web Management page (http://ip_address/RetainServer)
- Log in as Administrator or as a user with Administration rights
- Under the Configuration section on the left side navigation pane click Module configuration
- Select BlackBerry under Unconfigured Modules
- There are two tabs within the BlackBerry Module configuration.
Core Settings - To enable the BlackBerry Module, select the ‘BlackBerry Module’ from the module page and click ‘configure’. The BlackBerry module needs to be enabled on this page to make it active in the Retain system. Before data can be archived, the BES log file path must be provided. The log file location must be the address to the log files from the Blackberry Server. Regardless of where the worker is installed, local or remote to the Blackberry Server, it must have direct file access to the logs. This access can be accomplished through a shared folder or location, but it must be available to the Retain worker. Appropriate access through firewalls or network security must be made available if the worker is not installed on the BES system.
BES Server - This portion is designed to configure communication with the BlackBerry Server for the Address Book Sync. Server Name - The server name can be anything you would like. It is good practice to name this something unique to the BES Server its configured for Logfile Path - The path to your BlackBerry Enterprise Server log files. For this portion the logfile path is not verified SOAP Host - This is the IP address of Host name of the BlackBerry Enterprise Server SOAP Port - BES 5: 443 | BES 10: 38443 | BES 12: 18084 SOAP User - An Administrative user on the BlackBerry Administration Service SOAP Password - Password for configured user
SMTP Connector - In order to allow the forwarding of BES items out of the Retain system, the Blackberry Module needs to be configured with an SMTP connector. This connector will provide an outlet into the mail system, for Blackberry data to be forwarded if desired. The amount of configuration necessary for the SMTP Connector depends on the specific setup of the SMTP system. If a username and password are required for the host SMTP system, then it must be provided. If it is not required, the configuration of the username and password is optional. The address of the Mail Server, the SMTP Mail From Address, and the SMTP Mail To Address, must all be configured for the system to function correctly. The ‘to’ address is the destination address where the data will be forwarded to. The ‘from’ is the specified ‘from’ address. In addition, the ‘Enable SMTP Connector’ option must be enabled in the‘jobs’ configuration for the Blackberry section.
[edit] Schedule
Schedules define WHEN to perform the data collection as defined in a JOB. A Job defines what SCHEDULE, PROFILE and WORKER to use.
-You can create as many schedules as you need. -Schedules have no functionality until they are associated with one or more JOBs. -They can be shared across multiple JOBs. -You cannot DELETE a schedule that is currently associated with a JOB. -Schedules can be recurring or one-time.
SINGLE schedules are configured with one start time, and one date.To keep jobs from queuing up to the worker, select the box to do not resend a job that has an elapsed start time. Otherwise, a job will queue up on the worker if a previous job has not yet completed, causing the new job to start as soon as the current job is complete or terminated.
RECURRING schedules are configured using a start and interrupt time, and which days to start on. Choose when on which days you want this schedule to trigger a job. You may specify maximum job duration. If the job runs longer than the time limit you set, it will be interrupted. For cases where you do not want the job to be interrupted – for example, with a job you know will last longer than 22 hours, you may prevent the schedule from interrupting the job. Don’t forget to “Save Changes”.
[edit] Profile
A profile assigned to a job tells the system what to archive. Message types, status, date ranges, excluding or including specific users, private or public folders or both, and redundancy check settings are all configured under the profile. The profile defaults to collecting all data, with no date restrictions. After Retain has finished collecting data the first time, you'll want to narrow the date range and criteria considerably, or pay a high price in performance.
To create a new profile to archive BlackBerry data, first select the ‘add profile’ button and name the profile. Then continue configuring as desired.
Core Settings:
The Core Settings tab holds the enabling setting, which allows jobs connected to this profile to run. This must be checked before jobs will run.
Message Settings:
The Message Settings tab can be used to exclude or include specific data types for this profile. By default all types of messages will be archived. All BlackBerry data which is selected to be archived MUST be logged in the BlackBerry Enterprise Server or the data will not be available for Retain to archive it.
Scope:
The Scope tab dictates how pervasive the attached archiving jobs are, and what kind of duplicate checking is done, or not done, in the Retain system. The Date Range instructs Retain to scan for, and archive, messages after, or before, a certain date. This is useful if only specific chunks or areas of mail are to be archived. The duplicate check either scans for all messages, or instructs Retain to only look for messages which are newer than the Item Store Flag, which is a time-stamp on the system. All messages newer than the time stamp will be archived by Retain if the item store flag is utilized, but Retain will ignore all messages which are older. It is recommended to have at least the first job archive everything, with the item store flag checked to archive everything possible in the logs, while setting the item store flag. Subsequent jobs should use the duplicate check and the item store flag to only archive new data afterwards.
Advanced:
If you want to be more specific as to what to dredge or not to dredge, add the criteria here. Each line will be logically AND-ed together. Think “Dredge all items where the following is true:” Criteria A and Criteria B and Criteria C etc.
You may select based on:
-Subject -Sender -Recipient -Size -Attachment Name
Whether they are equal to, not equal to, whether they contain or do not contain the item you specify. This gives you great flexibility and granularity. It allows you to customize dredges and retention for many different groups, or even individuals.
[edit] Worker
The worker is the part of Retain that actually pulls the data from the message system servers. The Worker contacts the Retain Server and gathers information on which jobs are associated with it, when to start them, and which items to get. Workers can only run one job at a time. Workers are not module specific, but may require module specific settings. These settings, where applicable, are found under the module specific tab.
Worker Configuration From the Retain home page, click Workers. 1) Click “Add Worker”.
2) Put the name of the worker in the “Worker” box. Example: “BES Worker 1”.
3) Check the Connections Tab to be sure the Retain Server IP address settings are correct This tells the Worker where the Retain Server is located. Server Host Name or IP address must be correct, this should be populated automatically but double check for accurace. The server Connection setting must be able to resolve to the Retain Server if placed in a browser address bar. The connection tab holds very importand contact information for the worker's connection to the Retain Server. Make sure that the Retain Server connection address is correct, or the Retain Worker will not be able to talk to the server.
4) The Module Specific tab contains information specific to individually configured modules. The settings specified here are tied only to the specific worker configured, and does not affect the rest of the system. In General this setting may be ignored unless required. If required, input the appropriate information.
5) After you have checked and set your settings, click “Save Changes” before browsing to another page to finish creating and to save your worker.
After the worker has been saved, two additional tabs become visible for settings on the worker. You MUST save changes on a newly created worker before the ‘Bootstrap’ tab is available to be used. The status tab for the new worker will not hold any pertinent information until a job has been run or the worker has registered to the server and is in communication. Once a job has been completed, information will be displayed on the general status of the selected Worker.
6) After you have saved the changes, the Bootstrap tab appears (see below). Click “Download File” to save the configuration to a local workstation.
7) Save this file to your hard drive in a place where you can find it. Click “Save”.
8) Open a new browser window and type: http://< your-RetainWorker-ip >/RetainWorker
When a worker has not been initialized a screen will appear asking you to upload a bootstrap file.
9) Upload the bootstrap file.
-Click “Browse” to find the file you just downloaded to select it -Click “Open”. -Click “Upload File”. -If the upload went well, you will see a message that says "Configuration Complete. From now on, you'll have to log in as a user (such as the initially created admin user) with the Manage Workers right:
After you click the “Continue” button, you will be returned to the login page.
10) Login as Admin. 11) Verify the Retain Server connection success by selecting the Server Connection tab.
[edit] BlackBerry Job
Blackberry Jobs All data collection for the Blackberry module is configured through the Blackberry job interface. The jobs here combine the Profile, the Schedule, and a Worker together to archive the specified server logs. A Job must have a Profile, Schedule, Worker, and a selected server, before it can be saved or run. To create a new job, click, ‘Add Job’, and configure and save.
Core Settings The core settings tab contains the basic configuration. Select the ‘Job enabled’ option to enable archiving. Also select a Schedule, Profile, and Worker to complete the job. If the SMTP Send is enabled, the worker will forward BES message data
The Data Expiration is an option to place a timestamp on the data in the Retain database, which allows for ease of automation for the deletion manager. In addition, devices such as NetApp, Centera, Caringo, and Hitachi HCAP may use this number to enforce hardware level protection of the stored item so that no one (including Retain) may delete the item before its expiration date. Job Expiration is not retroactive for mail in the database, and only applies to mail archived by the job that it is active for. The base folders and criteria are specified under the custom expiration dates accessed through the ‘Add’ button at the bottom. All messages included in any specified folders will have a different date or be exempted from the standard expiration date. In order to have messages with custom job or folder expiration dates properly expire, the deletion management date scope must be set to delete messages with an Expiration Date older than 1 day.
Mailboxes The mailboxes tab contains the target users, and BES servers to be archived by the selected job. Under the Mailbox tab, select the BES server to be included in the Job. The list of available BES servers is configured from the Modules page. Once selected, all users associated or present on the BES server are included. Once a job has been run, the user list will be populated and further configuration is possible. The job may also be restricted to specific users, specified on the ‘Include Users’ list, or restricted to exclude users listed on the ‘Exclude’ list. To add users, expand the User’s option box and select the users to be archived. Users may be selected through the user search interface, spawned through selecting the ‘add user’ button. If there are users listed, then the users listed in the ‘Include Users’ box are the ONLY included users, while those listed in the ‘Exclude’ window are excluded from the jobs.
Notification All errors, summaries, and statistics may be sent in an email notification to a desired address on job completion. If it is desired to be notified, configure the notifications under this tab. At least one of the two notification options must be selected in order for an email notification to be created.
Status On initial creation, there will be nothing of interest shown under the ‘Status’ tab, as the job is under creation or has not yet run. However, after a job has been run, or has begun, the ‘Status’ tab becomes a monitoring tool to report on exactly how much mail and mailboxes have been archived, along with any errors encountered.
[edit] How It Works
[edit] BBM Protected Overview (Plus Many Other Topics) Training Video
Below is a video recording from July 31, 2015 given by our product manager, Colin Bretagne. It requires a video player that can play a .webm file. You can download one of those players here: VLC Video Player
Click on this link for the training video: Video of training given my Colin Bretagne July 31, 2015
[edit] Installation and Setup
In order to archive for BlackBerry certain steps need to be in place
- The users device needs to be in Regulated Mode
If the user device is not set to regulated mode the information will not be available for archive
- The BlackBerry Enterprise Server needs to provide log files for Retain to Archive from.
Retain will Dredge the log file created by the BlackBerry Enterprise server in order to gather the information it needs for the Archive.
- The BlackBerry Module needs to be configured on the Retain Server
Configuring the Blackberry module is done in a few short steps. - Enable the job. If the enable job button is not checked no jobs will be sent to the Retain Worker - Blackberry Enterprise server configuration. You can name the server anything you would like. The log file path is critical to the success of a job. The location of the Blackberry Enterprise log files is entered. The location must be either an NFS share or a Samba Share.
- A Schedule, Profile, Worker and Job need to be created on the Retain Server.
- The schedule determines how often the job will archive new data - The profile allows you to set what type of messages you will archive, what scope/range of messages will be included, etc - The worker is responsible for the communication between the Retain Server and the Blackberry Enterprise Server. - The Job is where all the components come together. You will chose the schedule, profile and worker that you want to use. You can also select what mailboxes you want to archive. You can archive an entire system or specific users
[edit] Archive Display and Interaction
Archived information is displayed within the users 'store' according to category. General information about the message/phone call is displayed on the main screen. Such as the From, Recipient, subject and date.
Opening the message will provide specific details about the message. You will be presented with a Message tab and a Properties tab.
-The message tab will show you who the message is To and From, the subject and will show you the contents of the body of the message. -The properties tab gives specific information - Item Type - Type of communication (BBSMS, BBMMS, etc) - Box Type - Tells you if the message/phone call was sent or received - Owner UID - Users Blackberry PIN - Read - if the message was opened or not - Confidential - Indexing - Tells you if the message has been indexed for searching - Created - Created date - Delivered - Date Delivered - Stored - Date stored in Retain - Expiration Date - Date set for deletion if deletion management has been setup in Retain - Begin Date - Shows time lapse of message or phone call - End Date - Shows time lapse of message or phone call - Path - Give the directory path for the message/phone call
Additional Properties
- postoffice - Post office that holds/held the message - email System - domain - internet - abSystem
Audit
- Information under this section will show an audit trail. Such as when the message was archived, the IP address that archived it.
[edit] Blackberry logging
BES Log Reading Tips
The logs for BES data will be in the RetainServer log
[edit] Dev overview
The Retain AddressBook sync was written a few years ago using SOAP to connect to a BES10 server. It has been tested with a BES5 and BES12 system since then using the same code with slight modifications. This code was written using the BES10wsdl.wsdl and BES10Utilwsdl.wsdl that I believe came from BlackBerry.
The administrator configures the BES credentials in Retain (host, port, username and password). The Retain AddressBook code will use this information to make a SOAP connect from the RetainServer to the BES server. The url being used would end up being something like:
https://bes10.qa.gwava.com:38443/enterprise/admin/ws
The code starts by making a request for GetSystemInfoRequest information. The getSystemInfo() call requesting the metadata (REQUEST_METADATA) looking for the "BWS Version" is for logging purposes but also to understand how to interpret the values being retrieved from a BES5, BES10 and BES12 system.
The code then sets up a request for the GetUsersRequest list. The GetUsersSortBy is set up with the setDISPLAYNAME(true) and the setValue("DISPLAY_NAME"). The getUsers() request is called with the setSearchCriteria and the SetSortBy being set. The setSortAscending(true), setPageSize(100) and setMetadata(REQUEST_METADATA) are configured. At this point the getUsers() response generates a user list in memory, wanting to fill in some extra information.
In order to get more information, the code then sets up a request (url admin/ws) for the GetUsersDetailRequest details. The setMetadata(REQUEST_METADATA), getLoadUserEmailProfiles(true), getLoadAccounts(true) and setLoadDevices(true) are configured. The code adds in each user returned from the user response in order to request the individual's details (.add(user)). At this point the getUsersDetail() response generates a list of details.
First - the DisplayName, FirstName, and LastName are pulled out.
Second - a list of email addresses is retrieved using the getDirectUserEmailProfiles() call, tagging the first email found using the getEmailAddress() as the primary email address and the corresponding userId found using the getUsername() or the getAccountName().
Third - a list of accounts is retrieved using the getAccounts() call, tagging (if needed) the first account found using the getEmailAddress() as the primary email address, the corresponding userId found using the getMailboxDN() and the corresponding userConfigId using the getUid() value (everything after the last hyphen "-" in it for future usage).
Finally - a list of devices is retrieved using the getDevices() call, storing all the BB Pins found using the getBlackBerryPIN() and the PhoneNumbers using the getPhneNumber().
If no primary email address and no device information is found, the user is ignored.
[edit] BES Address Book Sync
For the address book you’ll want to pull out the lines with [AddressBookSyncThread], just to make it easier to handle.
The address book sync will start with
01:02:05, 663[AddressBookSyncThread] [INFO ] SyncABookJob: Run address book sync for blackberry 01:02:05, 663[AddressBookSyncThread] [INFO ] SyncABookJob: addressBookSync: com.gwava.address.sync.BlackBerryAddressBookSync 01:02:05, 663[AddressBookSyncThread] [INFO ] BlackBerryAddressBookSync: Starting BlackBerry Address Book sync ... 01:02:05, 788[AddressBookSyncThread] [DEBUG] BlackBerryAddressBookSync: Getting SOAP configuration for bizcorp.com 01:02:05, 788[AddressBookSyncThread] [DEBUG] BESAddressBook: connectToServer - connecting to <server named in module> at <IPaddress or hostname>:<port>(usually 80 or 443)
Getting User List, emails and PIN or Phone Numbers When the BES AddressBook lookup occurs you should see the following log for each user it finds:
"getUserList - collecting details for user", followed by displayName, followed by BESUid. A BES 10 user will have a 4-6 digit number, a BES 12 user will have an alphanumeric GUID Something like: BES 10
01:03:02, 500[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, John, 32621
BES 12
07:19:57, 661[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Roe, David, 6388607a-1235-4347-85d1-652070e46d85
Then it looks in the BES email profiles for the email address. No direct logging of this action.
Then it looks a second time for email address in the BES Accounts. No direct logging of this action.
Then it looks in the BES devices for the pin and the phone number. No direct logging of this action.
If it finds nothing in the devices section (meaning the pin and phone number) it logs:
"getUserList - ignoring user", followed by display name, followed by "phone number and PIN not available".
And no user and device will be created or updated in the device management table.
If it finds something then it logs:
"getUserList - BES 10 or 12 generated nameId", followed by the nameId
Otherwise it logs:
"getUserList - ignoring user", followed by display name, followed by "email address not available",
Then nothing should be created or updated in the device management table.
Examples: If a user is found with a Display Name and email address and PIN or Phone# for a device, we’ll see:
01:03:02, 500[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, John, 32621 01:03:03, 436[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - BES 10 or 12 generated nameId bizcorp.com/2125551212
If a user has a Display Name and Email but no device, we’ll see:
05:37:31, 668[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Doe, Jane, 33234 05:37:31, 668[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - ignoring user Doe, Jane, phone number and PIN not available
If a user has a Display name but no email, we’ll see:
05:07:01, 640[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - collecting details for user Person, Person, 4447 05:07:01, 640[AddressBookSyncThread] [DEBUG] BESAddressBook: getUserList - ignoring user Person, Person, email address not available
The address book sync will end with
09:15:39, 536[AddressBookSyncThread] [DEBUG] BlackBerryAddressBookSync: Refreshing the device cache 09:15:39, 536[AddressBookSyncThread] [DEBUG] RetainDBStorageEngine: Priming the device cache by reading in the list of devices 09:15:46, 634[AddressBookSyncThread] [DEBUG] RetainDBStorageEngine: Loaded 101628 device(s) into the device cache 09:15:46, 634[AddressBookSyncThread] [INFO ] BlackBerryAddressBookSync: BlackBerry Address Book sync finished 09:15:46, 649[AddressBookSyncThread] [INFO ] SyncABookJob: AddressBook job completed
[edit] BES Message Archiving
You’ll want to pull out the message storage thread listed with "[REST-Default-" There are three kinds of messages: BBPIN (text PIN messages), BBSMS (text SMS messages), and BBMMS (multimedia MMS messages)
An individual message will look something like this:
00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Checking authorization with com.gwava.restapi.security.auth.application.OneRightApplicationAuthorizer@73701a09 for the application Retain Worker 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Application Retain Worker successfully authorized. 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: Checking authorization with AuthorizerDisjunctionChain [authorizers=PrincipalTypeAuthorizer [requiredType=Token] OR PrincipalTypeAuthorizer [requiredType=Collector]] for principal RetainCollectorPrincipal [primaryUID=B7CD9A6307A4A9903E2BC3AC0013D87A, secondaryUIDs=UIDSet [uidSet=[]], aclSet=UserACLSet [aclSet=[ADMIN_MANAGEWORKERS, USER_DELETEOTHER, USER_EXPORT, ADMIN_SEECONFIDENTIAL, USER_FORWARD, ADMIN_MANAGEUSERS, ADMIN_GLOBALTAGDEF, ADMIN_AUDIT, ADMIN_DELETE, ADMIN_MANAGESERVER, ADMIN_SEARCH, ADMIN_GROUPTAGDEF, USER_AUDIT, ADMIN_PUBLISH, USER_USERTAGDEF, USER_PRINT, ADMIN_ACCESSRM, USER_READWORKER, USER_SELFCONFIDENTIAL, USER_APPLYCONFIDENTIAL, ADMIN_RESTORE, USER_ATTACHMENT, USER_DELETEPERSONAL, ADMIN_LITIGATE, USER_RESTORE]]] 00:01:43, 359[REST-Default-3357] [TRACE] AuthorizedProcessor: User B7CD9A6307A4A9903E2BC3AC0013D87A successfully authorized. 00:01:43, 359[REST-Default-3357] [TRACE] PreauthorizedProcessor: Authorization succeeded; calling performOperation 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: InsertArbitraryMessagesProcessor.performOperation called 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: We will attempt to archive 1 messages now. 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: We also have 1 bytestreams to process. 00:01:43, 359[REST-Default-3357] [TRACE] InsertMessageProcessors: InsertArbitraryMessagesProcessor.performOperation exit 00:01:43, 359[REST-Default-3355] [TRACE] InsertMessageConverters: NewMessageToArchiveEmailParams.convert called for DEB77E29C1C4E84B4C86597428E6E557 00:01:43, 359[REST-Default-3355] [DEBUG] FolderPathCreator: getOrCreate Folder started. 00:01:43, 359[REST-Default-3355] [DEBUG] CacheProvider: Folders - Looking for element with cacheKey : 86C69E078AF2B44055FBE013F7D37CFE 00:01:43, 359[REST-Default-3355] [TRACE] CacheProvider: Inserting element with cacheKey : 86C69E078AF2B44055FBE013F7D37CFE 00:01:43, 359[REST-Default-3355] [DEBUG] CacheProvider: Folders - Looking for element with cacheKey : BE3B7391BADCAE2223DC45D755B690CD 00:01:43, 375[REST-Default-3355] [TRACE] CacheProvider: Inserting element with cacheKey : BE3B7391BADCAE2223DC45D755B690CD 00:01:43, 375[REST-Default-3355] [TRACE] FolderPathCreator: getOrCreate Folder ended, returning 3879 00:01:43, 391[REST-Default-3357] [TRACE] InsertMessageProcessors: MessageInsertionMapper: about to insert 1 messages. 00:01:43, 391[REST-Default-3357] [DEBUG] SingleProcessStorageEngine: receiving document Are you still up? .txt done. hash:20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB 00:01:43, 406[REST-Default-3357] [INFO ] RetainDBStorageEngine: About to commmit transaction null 00:01:43, 406[REST-Default-3357] [DEBUG] RetainDBStorageEngine: [HibernateUtil.insertMessage]: , msgID: DEB77E29C1C4E84B4C86597428E6E557, uuid: 3cdad518-e3d8-467e-9dc0-eadcfabe9517, doc: com.gwava.dao.social.Document id=644326, hash=20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB 00:01:43, 406[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Looking up device by phone number 2125551212 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Message was present: Are you still up? , doc: com.gwava.dao.social.Document id=644326, hash=20108D34CD4D1FED86638BD427E0E279245119E26D7CF06DF3CA7961DF747ABB 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: TRANSACTION: skipping double attachment 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: Are you still up? - attachments hasChanged: false 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: "Are you still up? .txt" has parent: \doed\BlackBerry\BBSMS isSubmail: false has changed: false 00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: check if indexing needed... 00:01:43, 422[REST-Default-3357] [TRACE] RetainDBStorageEngine: Item 7575211 has not been added to index queue because it has been indexed. 00:01:43, 453[REST-Default-3358] [TRACE] SpringAsyncUtils: UpdateResultOnSuccess: Setting DeferredResult to success value: <200 OK,com.gwava.retain.rest.v1.insertion.impl.InsertMessageResponseImpl@7677d2a5,{}>
For a message, you can find the user by the nameID from the address book sync: For example if the users Display Name is David Doe and nameID is "doed"
05:27:28, 899[REST-Default-3453] [DEBUG] RetainDBStorageEngine: "Fw: Test1.txt" has parent: \ David Doe\BlackBerry\BBPIN isSubmail: false has changed: false
00:01:43, 422[REST-Default-3357] [DEBUG] RetainDBStorageEngine: "Where are you? .txt" has parent: \doed\BlackBerry\BBSMS isSubmail: false has changed: false
06:37:18, 458[REST-Default-3469] [DEBUG] RetainDBStorageEngine: "Can these to get any cuter?!?! I swear I'm comin.txt" has parent: \doed\BlackBerry\BBMMS isSubmail: false has changed: false