Difference between revisions of "Retain Deletion Management"

Latest revision as of 19:36, 27 October 2017

[edit] PAGE OWNER: Steve Dorrough

After reading Retain Administration and Users Guide on Deletion Management, try this hands-on exercise or refer to the section below it, "How It Works"

[edit] Hands-On Exercise

Prerequisite:  Archive mail from your test mail system a '''few days in advance'''.  The more mail the better, but at least have a few items.  This exercise will require that the items be in Retain for over two days.

1.  Login to your Retain server (the actual VM server, not the RetainServer web UI).
2.  Stop tomcat: rcretain-tomcat7 stop (Linux) or stop the Apache Tomcat service on Windows.
3.  Rename the "index" directory to "index.old".  The index directory is located under your Retain storage directory (see Server Configuration | Storage).
4.  Create a new index directory.  If on Linux, make tomcat the owner of that directory:  '''chown tomcat:tomcat index'''.
5.  Switch over to the server itself again from the Retain web UI and tail the current day's RetainServer log.  Baretail.exe can be downloaded and used on Windows servers (see [http://support2.gwava.com/kb/?View=entry&EntryID=527 Location of Logs].  For Linux, change to /var/log/retain-tomcat7 and type: tail -f RetainServer.[yyyy-mm-dd].log.
6.  Start tomcat.
7.  Login to the Retain Server web UI (http://[ip address]/RetainServer) as admin.
8.  Under the Management section in the left-hand pane, select '''Deletion Management'''.
9.  Under Core Settings, click on the checkbox for “Job Enabled” to enable it and select “Delete messages as they are processed.”
10. Under the Date Scope tab, click on the arrow for the drop-down box, “Delete messages where”.  Note the various options.  For this exercise, leave it at the default “Date Stored in Retain”, which is the date the items were archived.
11. While still under Date Scope, select type “1” in the “Older than ___” field and leave the drop-down set to “Days”.
12. Under the Job Members tab, click on the drop-down box below “Include these objects” and select the mail server on which you wish to run this deletion job. Then, click on the “Add Mail Server” button.
13. Go ahead and check the boxes to have mail notifications sent to your mailbox.
14. Under the Schedule tab, click on the drop-down box for “Run Job when” and select “manual”.
15. Click on the “Save Changes” icon at the upper-right of the screen.
16. Click on the "Run Job Now" button.
17. Switch over to the actual Retain server and look at the tail of your RetainServer log.  What do you see happening?  Did anything get deleted?
18. Login to a Retain mailbox and check to see if anything older than 1 day.  Was anything deleted?  Why not?
19. Logout of RetainServer web UI.
20. Go back to the actual Retain server and stop tomcat.
21. Delete the new index directory you created.
22. Rename index.old to index.
23. Start tomcat.
24. Login to the RetainServer web UI as admin.
25. Go to Deletion Management and, on the Schedule tab, click on the Run Job Now button.
26. Switch over to the actual Retain server and look at the tail of your RetainServer log.  Now what do you see?  Any deletion activity?
27. What is the role of indexes in a deletion job?  See Level 2 section to check your answer.

[edit] Troubleshooting Information

There are four main stages of a deletion job:

[edit] STAGE 1

• Query the indexes for all message IDs matching the date criteria.
• Delete the message records in the indexes for those message IDs that get returned.

[edit] STAGE 2

• Mark all of the message records in the database that qualify for deletion. It sets the folder field to the "dumpster" folder.
• Delete all message records that are in the "dumpster" folder. In the log, you'll see "Deleting node [message id]"

[edit] STAGE 3

• Delete the corresponding attachment records for those messages from the t_document table. This is shown as "DeleteDao: Deleting com.gwava.dao.social.Document..." in the Server log.

[edit] STAGE 4

• Delete the files off disk that correspond with the records in t_document that were removed. This is shown as " DeleteDocuments: deleteDocumentList:..." in the Server log.

It actually rotates between stage 3 and stage 4. Perhaps we should just call them both "stage 3", but one works in the DB and the other with the file system, so this explanation separates them logically. Stage 3 collects a list of files (hashes) to delete as it removes their references from the database. It probably limits the size of that list for memory and other purposes, so it does it one list at a time.

The following section will describe the deletion stages in more detail. With each stage, the log entries you should expect to see in the RetainServer log are shown. If the Server logging level is set to diagnostic, it will include all entries that apply to the situation (INFO and DEBUG); otherwise, it will just show INFO. Thus, each logging example will provide the conditions followed by the text (in quotes) that will display in the log. For example, the stage 1 shows the first log line as follows: If INFO: "Running deletion job". Thus, If INFO is the condition (the logging level set on the Server) and "Starting the deletion job" is the text displayed in the log.

[edit] STAGE 1 – The Deletion Job

[edit] 1. Deletion job is initiated.

If INFO: “Running deletion job.”
If DEBUG and if report only mode: “Will only report, not delete”
If DEBUG and not report only mode: “Normal deletion mode”
If DEBUG: “autoApprove” [Boolean value]
If DEBUG: “Deletion report stored at: [filePath]"
If DEBUG and PO chosen: “Deleting PO [PO]"

[edit] 2. Searches the indexes.

It starts by sending a query to the Indexer with the given scope and gets a total number of hits as well as the number of hits within the first page. A page consists of 500 records.

If DEBUG: "On Page # [pageNumber]"
If INFO: "Aborting early, deliberately since this is a trial" (if the last page number is less 20 and if in REPORTING ONLY mode)
          By default, we only report the first 10,000 items to be deleted 
          (500 items per page and 20 pages by default: 500 x 20 = 10,000)
If INFO: "Result page TotalHits: [total number of hits from index] [hits on current page] in result set"
[loglevel unknown]: "inSet: [list of all message IDs that are identified]"

It puts all those results in a list of items to be deleted from the index. The list is kept in memory and is lost if Tomcat is stopped at this point.

[edit]

As it queries the Indexer, you'll see this entry in the Indexer log. Note that the following example is from a deletion job where the date range was set to "1 day or older" and triggered off of delivered date:

IndexSearchMessageConsumerImpl - DeletionQuery:delivered:[* TO 20150226215106] AND (domain:[domain] AND postoffice:[post office])

From the text box above, you can see that it shows that it is being triggered off of the delivered date because it states "DeletionQuery:delivered". Since the date range was "1 day or older", the range of the query is shown as follows: * TO [numeric string]. The begin date of the range is represented by an asterisk because it goes back infinitely. The end date of the range is represented by a numerical string. The first 8 characters of the numeric string represent the day before the date this was run. I ran the deletion job on 2015-02-27. I do not know what the last 6 digits represent at this time.

[edit]

It then displays the total hits found:

 
IndexSearcher.search: +delivered:[* TO 20150226215106] +(+domain:steve +postoffice:parents), totalHits: 8144, localHits:1000, sort:<score>

[edit]

It appears that the database driver picks this up and displays all of the Retain message IDs from retain.t_message.message_id(?) after "inSet:"

[DeletionJobTask-1-thread-1] TRACE com.gwava.hibernate.HibernateStringUtil - inSet: [message IDs]

[edit]

... and then proceeds to list the first 500:

[DeletionJobTask-1-thread-1] INFO  com.gwava.deletion.AbstractDeletionTaskExecutor - List<EMail>: [message IDs]

[edit]

At this point, it is assumed that it sends that list to the Indexer with instructions to delete them from the indexes as we see these entries in the Indexer log:

IndexDeletionMessageConsumerImpl - Trying to process the operation: INDEX_DELETE
LuceneIndexingManager - Delete [message ID] from index
LuceneIndexingManager - Delete [message ID] from index'''
etc...

[edit]

It does this in batches of 500, as we see these log entries in the Indexer log every 500 items:

IndexSearchMessageConsumerImpl - Trying to process the operation: INDEX_DELETION_TASK_SEARCH
IndexSearchMessageConsumerImpl - DeletionQuery:delivered:[* TO 20150226215106] AND (domain:steve AND postoffice:parents)
LuceneIndexingSearcher - searching with offset: 0, count: 500
LuceneSearchController - query: delivered:[* TO 20150226215106] AND (domain:steve AND postoffice:parents) field: uuid
LuceneSearchController - query.toString: +delivered:[* TO 20150226215106] +(+domain:steve +postoffice:parents)
LuceneSearchController - 1425073871005, start query: +delivered:[* TO 20150226215106] +(+domain:steve +postoffice:parents)
LuceneSearchController - 1425073871014, end query: +delivered:[* TO 20150226215106] +(+domain:steve +postoffice:parents)
LuceneSearchController - IndexSearcher.search: +delivered:[* TO 20150226215106] +(+domain:steve +postoffice:parents), totalHits: 7644, localHits:1000, sort:<score>
IndexDeletionMessageConsumerImpl - Trying to process the operation: INDEX_DELETE

[edit] 3. Removes litigation hold messages

Removes all messages pertaining to users that are on litigation hold from the list of messages to be deleted.

If DEBUG: "[emailID] is under litigation hold and cannot be deleted"
If DEBUG: "[emailID] doesn’t belong to user"

[edit] 4. Updates each message record, putting it in a “deleted” state

This is known as putting the message record "in the dumpster".

There are no log messages for this step. This step is where it puts the messages in the "dumpster". The dumpster is a folder record in the t_folder table and is literally named "Dumpster".

You can find the messages waiting to be deleting in the t_message table by using the following query: SELECT * FROM retain.t_message where folder_id = 273;

NOTE: "273" is the folder_id from the t_folder table which you can get by executing this SQL query: SELECT * FROM retain.t_folder where f_name = 'Dumpster';

[edit] THE FOLLOWING INFORMATION WAS GIVEN TO ME BY MIKE BELL AND IS VERY SKETCHY - WAITING FOR MORE ACCURATE AND PIN-POINT EXPLANATIONS

5. Once the search of the index is completed (reference step #2), the indexer gets called to delete those items from the index. The time for this to occur varies depending on the indexer load and the scope of the deletion job. It could take minutes, hours, or weeks.

At this point, a browse of the archive mailbox already fails to show the item(s) because the item is now marked for deletion in the database.

Because the item is marked for deletion in the database and before the indexer removes it from the index, it is possible for someone to perform a search of the mailbox (which uses the index) and get back a hit; however, under this scenario, all that will happen is that in the search results the item will be shown as “---- this messages is queued for deletion ----” and nothing more.

The deletion job keeps looping through each page of the index until it is done. This happens while the INDEXER continues working on deleting items from the index. Once done with the list, that's the end of deletion job.

[edit] STAGE 2 – The Dumpster Thread

The "dumpster thread" is awakened by the end of the deletion job and looks for new items in the dumpster. The "dumpster" is the t_dsref.f_state field. It would have one of two values:

• 0 = normal
• 1 = queued for deletion

As it finds items with a state value of "1", it pages through them and deletes those message records in the database, the ones where the folder_id is set to the dumpster folder's ID. In the sample text from the server log, the number following "Delete" and "Deleting node" is the message ID:

15:54:00, 985[DumpsterThread] [TRACE] DumpsterDiver: Delete 3608803
15:54:00, 986[DumpsterThread] [INFO ] DumpsterDiver: Deleting node 3608803

Events that can trigger the dumpster thread are:

• Deletion job completion.
• Server boot up.
• Maintenance.

If the final document entry is orphaned (reference count=0), it calls the StorageEngine to delete the physical file. See “Storage Engine Process” for more information (below).

When "Dumpster Thread ending" appears in the server log, it just means its last pass found nothing left to process in the database. It does not necessarily mean that the physical file itself has been deleted yet by the storage engine.

[edit] Storage Engine Process

A StorageEngine is just a black box with a delete method attached. How it works and its speed depends on the implementation:

Retain 2.x and Before: Standard Storage Engine

StandardStorageEngine, which we used regularly before 3.0 simply finds the file and deletes it. Just a simple straightforward method.

This was changed in Retain 3 because a straightforward implementation is too slow. It would synchronously delete index pieces and DB pieces etc. All would be easy if delete, wait for delete, delete, wait for delete in a linear fashion but that locks up things for days.

Retain 3.x: DataStore_Process Storage Engine

DataStore does more complicated things. Internally it simply finds the references in t_dsref and marks them in the deleted state as mentioned in the "Dumpster Thread" discussion (above). It does nothing more.

At some point, another background thread specific to the DataStore looks for items in the deleted state. After performing various sanity checks, it removes the t_dsref entry and the storage engine deletes the corresponding file from disk.

[edit] Reading Deletion Logs

As you go through the Server log you can trace the progress of the job:

If you schedule a deletion job it will run during the maintainance cycle at 1am (by default):

2015-08-12 01:03:22,584 [Maintain] INFO com.gwava.jobs.MaintenanceJob - Execute deletion jobs

But if jobs are set to manual then you will see this message:

2015-08-12 01:03:22,593 [DeletionJobTask-1-thread-3] DEBUG com.gwava.deletion.DeletionTaskHelper - Skipping deletion job. Only manual trigger selected.

You can trigger a job manually by clicking on the "Run Job Now" button under the Schedule tab and you will see this in the logs:

2015-08-12 09:06:03,072 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.DeletionTaskHelper - Manual trigger of deletion job requested. Not checking schedule

And the job begins:

2015-08-12 09:06:03,072 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - Running deletion job. 2015-08-12 09:06:03,072 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - Normal deletion mode 2015-08-12 09:06:03,072 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - autoApprove:true 2015-08-12 09:06:03,074 [DeletionJobTask-1-thread-1] DEBUG com.gwava.deletion.DJDeletionOperation - Deletion report stored at /var/opt/beginfinite/retain/archive/Deletion8464290409683004469EYSZTWSUTFUWYFMRMHWHU.html

If you are running the job against individual users you'll see them listed:

2015-08-12 09:06:03,094 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.DJDeletionOperation - Include user scope: [4632DB80-0EA9-0000-9A73-667864633738]

Then it begins to build batches of 500 eligible items from the indexes:

2015-08-12 09:06:05,301 [DeletionJobTask-1-thread-1] TRACE com.gwava.hibernate.HibernateStringUtil - inSet: 823104,823105,...,824095,824096 2015-08-12 09:06:05,301 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - List<EMail>: 823104,823105,...,824095,824096 2015-08-12 09:06:05,373 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - fromIndex=0,toIndex=100 2015-08-12 09:06:07,100 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - fromIndex=100,toIndex=200 2015-08-12 09:06:07,583 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - fromIndex=200,toIndex=300 2015-08-12 09:06:08,393 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - fromIndex=300,toIndex=400 2015-08-12 09:06:09,427 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - fromIndex=400,toIndex=500 2015-08-12 09:06:10,913 [DeletionJobTask-1-thread-1] DEBUG com.gwava.deletion.AbstractDeletionTaskExecutor - deleteIndexDocs 2015-08-12 09:06:10,913 [DeletionJobTask-1-thread-1] DEBUG com.gwava.deletion.AbstractDeletionTaskExecutor - deleteIndexDocs, tl.size: 500

Once it has build batches of items to be deleted it sends the email (yes, we know. This is going to be changed to when the deletion job actually ends) and begins to remove items from the database:

2015-08-12 09:06:58,459 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.DeletionTaskHelper - Sending mail after deletion job... 2015-08-12 09:06:58,501 [DeletionJobTask-1-thread-1] INFO com.gwava.deletion.AbstractDeletionTaskExecutor - DeletionTaskExecutor ends 2015-08-12 09:06:58,508 [DumpsterThread] DEBUG com.gwava.deletion.DeletionJobFactory - Dumpster Thread beginning 2015-08-12 09:06:58,512 [DumpsterThread] INFO com.gwava.deletion.DumpsterDiver - Dumpster dive begins 2015-08-12 09:06:58,573 [DumpsterThread] INFO com.gwava.deletion.DumpsterDiver - Got these many children of dumpster500 2015-08-12 09:06:58,573 [DumpsterThread] TRACE com.gwava.deletion.DumpsterDiver - Delete 823039 2015-08-12 09:06:58,582 [DumpsterThread] INFO com.gwava.deletion.DumpsterDiver - Deleting node 823039

and continues until that is finished:

2015-08-12 09:09:26,605 [DumpsterThread] INFO com.gwava.deletion.DumpsterDiver - Delete orphaned documents 2015-08-12 09:09:26,605 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - Removing orphaned documents page

Then the job moves on to delete items from disk, which are also done in batches:

2015-08-12 09:09:26,692 [DumpsterThread] TRACE com.gwava.message.dao.DeleteDao - Deleting com.gwava.dao.social.Document id=841321, hash=560170B00B4B4C4FABA777D76D93BA98527C12E77AE77A6C2F7DB4580AF0FE8D 2015-08-12 09:09:40,482 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - Executed query in 13877 2015-08-12 09:09:40,489 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - deleteDocumentList:560170B00B4B4C4FABA777D76D93BA98527C12E77AE77A6C2F7DB4580AF0FE8D 0 841321

until it is done and the deletion job ends:

2015-08-12 09:12:30,777 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - Removing orphaned documents page 2015-08-12 09:12:30,777 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - Executed query in 0 2015-08-12 09:12:30,777 [DumpsterThread] INFO com.gwava.deletion.ng.DeleteDocuments - deleteOrphanedDocuments: Nothing left 2015-08-12 09:12:30,778 [DumpsterThread] INFO com.gwava.deletion.DumpsterDiver - Dumpster dive completed 2015-08-12 09:12:30,778 [DumpsterThread] DEBUG com.gwava.deletion.DeletionJobFactory - Dumpster Thread ending

[edit] Deletion Management - Retain 4.2

OVERVIEW
Retain 4.2 introduces a new Deletion Management system that has improved the effectiveness, reliability, and functionality from the old Deletion Management in previous versions. Customers in the past have had issues with Deletion Management with the following problems:
• Understanding how to configure, and launch Deletion Management
• Having the Deletion Management not delete all of the data, and not reclaiming the storage space that the customer needs.
• Creating orphaned documents, either due to indexing errors, errors within the code of Deletion Management, a restart of the tomcat service, or the server itself. These orphaned documents cannot be deleted, and causes customers to lose confidence in the reliability of the product.
• Reports of the Deletion Management only show 10k items to be deleted, which doesn’t accurately depict what the customer would like to see removed from their system. • When the Deletion Management runs it doesn’t reliably delete messages, and customers are stuck with messages that are still searchable where they should be deleted. This includes messages that go years back,

IMPROVEMENTS AND CHANGES
The new Deletion Management resolves all of the above issues. More importantly its design is to allow customers to delete their data easily using a friendlier interface that matches the archiving process, standardizing the look and feel of the software. At the same time customers ,and support, can be assured that the data will be reliably removed, and at any time if the Deletion Job is interrupted, it will pick up from where it left off. This prevents orphaned documents, and gives customers the confidence that their storage space will be reclaimed.

Listed below are the details the specifications and the improvements of the new Deletion Management.

Interface: Setup and Launching a Deletion Job

• The Deletion Management has a new interface mirroring the archive jobs menu. This was meant to standardize the look and feel for Retain, as well as help customers know how to set up when, and what to delete in a convenient format.
  
• The Deletion Management no longer runs during Maintenance, but instead runs with its own deletion schedule, and process which runs in its own background.
  
• The interface for Deletion Management is now contained under a new Data Removal section, which is only used for administration. Rights can be assigned to specified users and groups as necessary.
  

Deleting Messages
There are three ways to delete messages in Retain:

1. Mailbox Deletion:
This option allows administrators to delete ALL messages from users. It is intended to only run without a schedule, with a run now button as the only means to delete. A report can be generated prior to deleting messages processed.
The user(s) that are deleted from the Mailbox Deletion do not get removed from the address book in Retain. This will be added into future versions of Retain. In Retain 4.3 an option will be put into the Deletion Management to remove the user from the address book.
Customers can use the Mailbox Deletion to their advantage to remove users they no longer want to archive or have since left their email system. Once messages have been completely removed from the Retain system using mailbox deletion, the ts_store in the t_abook table is set to 0 for the mailbox and the user is no longer counted as an active license. It is highly recommended that the administrator removes the user from their e-mail system, or exclude them from archiving it altogether.

2.Item Deletion
The biggest improvement of the Deletion Management is being able to schedule a job, or multiple jobs and have them run without using the maintenance process in Retain. It is done by creating a schedule for each job (when you want to kick off the job), set your profile (what you want to remove), and creating a job , assigning the e-mail servers, or mailboxes. This gives customers flexibility, and can kick the job off very easily by adjusting the schedule. Future versions of Retain will have a run now button for the item deletion.

3. User Item Deletion
Users, if given the access rights by the administrator, can select messages from browse, or from their search in their archive and click to delete. Retain will delete the messages that are selected. This is best used for several messages. Large amounts of messages, or accounts, are best used with Item Deletion, or Mailbox Deletion.

[edit] Scheduling a Deletion Job

To schedule a job to run, create a schedule, profile and job, just like creating an archive job.

Choose the mailboxes in the job, select in the profile what to delete, and the date ranges. Then adjust the schedule to run at the desired time.

The Retain Server log will show when the job runs : DeletionTaskHelper

It will then state it is deleting indexes, then Deleting documents from the database, and then finally off of the physical storage.

[edit] Deletion Reports

The Deletion Management can create a report before deleting the messages as they are processed. These reports are stored in the /archive directory, and are named the date and time that the job kicked off. These reports will show what has been deleted. There is also a list of users and messages that are put in to a CSV file that will show the tracking of what has been deleted.
In previous versions of Retain the report would only show 10,001 messages maximum. This was a hassle for customers who wanted to know what would be deleted first, before they proceed with the actual deletion. Retain 4.2 fixes this, and provides the customer with a full report which the administrators can view and see every message that will be deleted for each account.
Notifications emails are also sent out, but only show basic information if the job has completed or not. Future revisions of Retain will show more detail in the reports, including the Mailbox Deletion report.. Also, future revisions will have a Deletion Report in the reporting and monitoring that will provide more information as well.

***Overall with the changes to the interface, and assigning the Deletion Management to run whenever the customer needs are major improvements to the product. Not only that, the system will reliably delete the messages without question, and the way it does it on the back end ensures that no messages become orphaned, or get missed during the process.

[edit] Deletion Management Deletion Process – Backend

When the Deletion Management runs, it will first run a query based on the criteria specified within the job. This query is dependent on the indexes.
Based on the profile, and the mailboxes selected in the job, the Deletion Management will flag the messages in the t_message table, altering the f_state record. If the last binary digit in the f_state record is a 1 or an odd number then that tells Retain that the message is flagged for deletion. The next step will then proceed with deleting the indexes, which as it does will create a log file in the /archive directory to track which users and messages it has deleted.
The deletion process will delete from the indexes first, then the database, and finally from the physical disk. This method also helps to prevent orphaned documents. As the items are flagged in the database they are tracked through the f_state record. Even if the indexes are deleted, but the metadata in the database is not, the Deletion Management will recognize this, and remove the messages. Even if the Retain Service is interrupted in any way, it will recognize the flags and will delete based on where it left off. This prevents orphaned items.
If the Retain service is ever interrupted restart the schedule and it will pick up from where it left off and delete the messages successfully.

The following are more details with the backend processes on each section of how the Deletion Management removes messages:
Index Process – Backend
When a mailbox deletion or a schedule, profile, and job is set to delete messages, the Deletion Management will run a query to see which users, and items need to be deleted. It is important that the indexes are functioning properly, and if not then a rebuild of the indexes is necessary.
Once the indexes are queried the Deletion Management creates a CSV file of the users and items that are to be deleted. This file is stored in the /archive directory with the date and time that the job ran. The query will proceed to flag the messages that need to be deleted. As mentioned above, it will modify the f_state record in the t_message table. Once it is flagged, it will remove the message.

*Note: Once a message is flagged to be deleted, there is no way to reverse it. Customers need to be sure that they know exactly what they are deleting before removing any data from Retain. A backup of the data, and the database, is required if they are needing to restore the deleted message.
 

Once the query is finished, the Indexes will be removed.

Item deletion psuedocode

• When job starts:

1. query index and get the list of eligible items for deletion
2. go through them page by page (each page = 500) --> EnhancedDeletionTaskExecutor.deletePage();
   1. find records in DB 100 by 100
      1. mark them for deletion
      2. assign each message to dumpster node
         1. if this message has a message parent, also assign it dumpster node
         2. if this message has children assign them to dumpster node. also append them to the list of index IDs for deletion
      3. commit transaction every 10 message
   2. delete items from index

• When Dumpster starts: DumpsterTaskExecutor.removeFromDumpster()

1. delete messages from DB : processAndDeleteMessagesFromDB(), deleteMessage()
   1. MessageProperty
   2. MessageRecipient
   3. MessageAttachment --> this will decrement refCount --> orphaning the document for the message (if no other message is referring to it)
   4. LegacyID
   5. MessageTag
   6. Message
2. delete documents : deletionOperations.getDeleteDocuments().deleteOrphanedDocuments()
   1. fetch top 1000 documents
   2. delete them from Disk
   3. delete them from DB
   4. continue until nothing is left

Database Process- Backend The next process in the data removal is deleting the entries in the database. Once the indexes are deleted, Retain will proceed to delete the metadata, still using the flags as its measure to delete the data.
The f_state is looked at then the data is removed.

The following is a more detailed sketch of what the f_state looks like and how the Deletion Management would delete messages:

Here are the flags for f_state in t_message
STATE_DELETED=1; // Item is in Deleted State
STATE_LITIGATIONHOLD = 2; // Item has been Litigation Held (not in a deleted state)
STATE_READ = 4; // Item has been Read (not in a deleted state)
STATE_UPDATE=8; // Index should UPDATE instead of ADD - eg atomically remove old documents associated and then add (not in a deleted state)
STATE_SUBJECTHTML=16; // SUBJECT (or other pieces) may have encoded HTML (not in a deleted state)
STATE_SUBJECTHTML=17; // SUBJECT (or other pieces) may have encoded HTML (message set to be deleted)
STATE_ATTACHMENTS = 32; // Has Attachments (not in a deleted state)
STATE_HASTAG=64; // Has Message Tags (not in a deleted state)
STATE_CONFIDENTIAL=128; // Is in Confidential State (not in a deleted state)
STATE_CONFIDENTIAL=129; // Is in Confidential State (not in a deleted state) (Message set to be deleted)

*Besides the STATE-DELETED=1 meaning the message is to be deleted, the binary could be a different number depending on the type of message. If the message contains an attachment, for example, then the number is not going to be a 1. When it is to be deleted, the first bit on the  binary number will change to a one, making the binary number an odd number, and be flagged for deletion.
 

• Example database change with message flagged:

44 = 00101100 --- This message has READ, UPDATE and ATTACHMENTS flags on
Set for deletion would change the number 44 to 45 = 00101101

*Another thing to keep in mind, that as the deletion management runs it will also check the f_referenceCount and set it to 0 in the t_document table. This lets the deletion management know to delete from the database as well as the phsycial disk. When the deletion management runs it will check the f_referenceCount, and if any of them equal 0, it will delete them from the disk as well as the t_document table. This also helps to prevent orphaned documents.
 

Physical Disk Removal – BackEnd As mentioned, when the f_referenceCount = 0, then it will delete the remainder of the data from the t_document, as well as from the physical disk.
The physical disk removal takes the longest. As the data is deleted, it will remove the .dat files from the hash, and will reclaim the disk space for the user.

[edit] Deletion Management Logs

The Deletion Management logs are still contained within the Retain Server logs. In future revisions of Retain they will be separated to their own logs.
To see that the Deletion Job is running look for this line:

DeletionTaskHelper-executor

With the three phases that the Deletion Management goes through to delete data from the Retain system, the logs will show these phases distinctly and clearly. The logging has been modified from previous version to be more clear on exactly what Retain is deleting. The following are examples with a brief expository on what to see in the logs during a Deletion Job:

Index Deletion:
When Retain gets a list of the data to delete via the indexes it will show the following in the log:

12:29:32, 052[RTSQuartzScheduler_Deletion_Worker-1] [INFO ] DJDeletionOperation: When processing job: Generate a report but don't delete messages.
12:29:32, 052[RTSQuartzScheduler_Deletion_Worker-1] [INFO ] DJDeletionOperation: Querying indexes, recording in memory the message IDs of records that match the deletion criteria.
12:29:32, 095[RTSQuartzScheduler_Deletion_Worker-1] [DEBUG] DJDeletionOperation: No include user scope
12:29:32, 096[RTSQuartzScheduler_Deletion_Worker-1] [INFO ] DJDeletionOperation: Deleting PO [GodsDom.greek]
12:29:32, 096[RTSQuartzScheduler_Deletion_Worker-1] [TRACE] DJDeletionOperation: initializeSearch: offset=0
12:29:32, 096[RTSQuartzScheduler_Deletion_Worker-1] [INFO ] DJDeletionOperation: On Page #0
12:29:32, 174[ActiveMQ Session Task] [TRACE] IndexService: Deletion query: (domain:godsdom AND postoffice:greek)
12:29:32, 175[ActiveMQ Session Task] [TRACE] IndexService: Deletion filter query (fq): storeDate:[1980-01-01T19:29:32.000Z TO 2017-10-26T18:29:32.000Z]
12:29:32, 466[RTSQuartzScheduler_Deletion_Worker-1] [INFO ] DJDeletionOperation: Result page TotalHits: 2061 500 in result set
12:29:32, 468[RTSQuartzScheduler_Deletion_Worker-1] [TRACE] HibernateStringUtil: inSet: 266,269,272,275,463,466,469,472,1793,1796,1799,1802,1805,1808,1811,1814,1817,1820,1552,1555,1558,1561,1564,1567,1570,1573,1576,1579,1582,1585,1588,1591,1594,1597,1600,1603,1606,1609,1677,1680,1683,1686,1689,1692,1695,1698,1701

Mailbox Deletion:
This is what you will see when performing a Mailbox Deletion:

12:27:15, 528[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: DeletionTaskExecutor begins
12:27:15, 528[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Running deletion job.
12:27:15, 529[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Normal deletion mode
12:27:15, 529[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: autoApprove:true
12:27:15, 571[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: start full mailbox deletion of: 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:15, 626[DeletionJobTask-1-thread-2] [DEBUG] AbstractDeletionTaskExecutor: deleteIndexDocs
12:27:15, 627[DeletionJobTask-1-thread-2] [DEBUG] AbstractDeletionTaskExecutor: deleteIndexDocs, SearchResults.size: 618
12:27:19, 325[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Deletion of messages has been done for mailbox: 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:19, 325[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Now deleting folders...
12:27:19, 358[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Deletion of folders has been done for mailbox: 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:19, 436[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Mailbox deletion has been completed for: 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:19, 455[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Looking for any mailbox that does not have archived messages and is mistakenly defined as active...
12:27:19, 455[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Successful session retrieval
12:27:19, 470[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Returning results: [247BB500-09CE-0000-B067-6F6464663663, 4885C901-09C3-0000-B067-6F6464663663, 5472E700-181B-0000-B067-6F6464663663, 7F196B80-182C-0000-B067-6F6464663663, 81219600-0922-0000-B067-6F6464663663, 83CFCA00-09BA-0000-B067-6F6464663663, 850F4C80-182C-0000-B067-6F6464663663, 850F4C81-182C-0000-B067-6F6464663663, 87E70F00-0AC2-0000-B067-6F6464663663, 88F88F00-02BF-0000-B067-6F6464663663, 8D445980-0AC2-0000-B067-6F6464663663, 903F4A00-0AC2-0000-B067-6F6464663663, 933A3A80-0AC2-0000-B067-6F6464663663, 94823080-0925-0000-B067-6F6464663663, 96352B00-0AC2-0000-B067-6F6464663663, 9B927580-0AC2-0000-B067-6F6464663663, 9E8D6600-0AC2-0000-B067-6F6464663663, A0EFC000-0AC2-0000-B067-6F6464663663, A4834700-0AC2-0000-B067-6F6464663663, AD467D80-1823-0000-B067-6F6464663663, AD467D81-1823-0000-B067-6F6464663663, B9CE6900-093C-0000-B067-6F6464663663, CE2F3A80-0844-0000-B067-6F6464663663, D36BC100-09CD-0000-B067-6F6464663663, E1193C00-0924-0000-B067-6F6464663663, FEEEAB81-09CD-0000-B067-6F6464663663]
12:27:19, 510[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: deleting AddressBook entry with uuid = 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:19, 725[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: delete UUID Mapping: 128 - 60beee8d-1f91-42a6-aec7-e8df19719921
12:27:19, 812[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: DeletionTaskExecutor ends

Database Deletion:
When seeing messages being delete from the database, the lines will show something like this:
12:28:03, 412[DumpsterThread] [INFO ] DeleteDocuments: deleteDocumentList:1B55E8253EB9B323B01A9773A452A30550F7665F4E4991BF133C263553A2B607 0 3604
12:28:03, 418[DumpsterThread] [INFO ] DeleteDocuments: deleteDocumentList:7F1C99CFAA51FB93E646F915979756493993148F431B02ACEB711975932FB209 0 3605
12:28:03, 421[DumpsterThread] [TRACE] DeleteDao: Deleting document entry from DB com.gwava.dao.social.Document id=4794, hash=E7FEF0A842443BDA08A6F1B561EC79DCCBB118796189D8D8EE2414FB255AE66F
12:30:09, 920[BackgroundDeletionThread] [TRACE] BackgroundDeletionThread: Delete DB Reference (large blob) for 4495 DEAB395DE11294E4206F6851DE565C464CC655AD4B0A1DB5D4F42389BDD9E47E

Disk Deletion:
12:30:09, 784[BackgroundDeletionThread] [DEBUG] BackgroundDeletionThread: Removing blob from large blob store with hash of 1960766F46190356631CF7C2A2C772DFABFFA14F8E17E3F50FE6655BDBDCD852
12:30:09, 785[BackgroundDeletionThread] [TRACE] BackgroundDeletionThread: DataStoreBlobID [id=FDE7E382A645559F7274581DA4DFD1C50BF6E6F308E3C56CE6A4D26AFEAD38AB] has 1 entries...
12:30:09, 785[BackgroundDeletionThread] [DEBUG] BackgroundDeletionThread: Removing blob from large blob store with hash of FDE7E382A645559F7274581DA4DFD1C50BF6E6F308E3C56CE6A4D26AFEAD38AB
12:30:09, 786[BackgroundDeletionThread] [TRACE] BackgroundDeletionThread: DataStoreBlobID [id=A91682FEF9CB63F6DCA3DCC29EE237F2A430E2FC7BAEEE5E01A186C82AF87D50] has 1 entries...
12:30:09, 786[BackgroundDeletionThread] [DEBUG] BackgroundDeletionThread: Removing blob from large blob store with hash of A91682FEF9CB63F6DCA3DCC29EE237F2A430E2FC7BAEEE5E01A186C82AF87D50

For more information on reading the Retain Server and Deletion logs see this KB article:

http://support.gwava.com/kb/?View=entry&EntryID=2850

To see if the Deletion Job has finished look for this line:

DumpsterDive - End

Upcoming for Retain 4.4 Deletion Management These are the list of items that will be included in Retain 4.3.
• Mailbox Deletion - create a report for all messages deleted for the accounts and put into a CSV file.
• Deletion Management having its own log, separate from the Retain Server log.
• Having a report in the Reporting and Monitoring for Deleted items
• Email CSV files, and detailed reports to administrator or other.

• Be able to remove old orphaned documents and meta-data (separate utility) • Status tab that will show the details of the job and to abort if necessary.
Future revisions of Retain will eventually contain a dashboard that will show the process of the deletion, when it is completed. As of right now the log files are still the only way to determine if a deletion job is running, and when it finishes.

[edit] Deletion Management Queries

The following are a list of queries that can be run to help identify the flags and to help troubleshoot database related issues with the Deletion Management.

https://traininggwava.microfocus.net/index.php/Retain_Database#Deletion_Management_specific_Queries