Deletion Management

Deletion Management is the most dangerous part of Retain since it removes data.

The main reason customers use deletion management because the data has reached the end of the retention period. This also saves on disk space.

Many organizations have a data retention policy, which defines how long it will hold data for. Once the message has reached the end of the retention period deletion management can be enabled to remove them from the index, database and archive.

Core Settings

• Job Enabled: This is disabled by default because we have no idea what the data retention policy of the organization is.
• When processing job: there are two choices and one of them doesn't even delete anything.
  • Generate a report but don't delete messages [1]
  • Delete messages as they are processed

Basic Options

Here you can select specific types of items that can be deleted. However, if all items are unchecked, then all items are eligible for deletion.

• Item Type
• Item Source
• Message Status

Date Scope

You can select the date range to delete items from Retain.

• Date Stored in Retain (This mostly works except for the all the messages that existed before the initial dredge.)
• Delivered Date (this is the best choice as it is the more real world friendly.)
• Expiration Date (this can be set in the job when an item is stored.)
• Created Date (this can be spoofed by the sender so cannot be trusted.)

Job Members

Like any job it needs something to act on.

• Mail Server (this will effect an entire mail server)
• Users (just individual users)

Notification

Set the parameters for sending the deletion report. In Retain 4 this message is sent when the job is completely done.

Schedule

• Run Job when
• Run Job Now

Mailbox Deletion

Deleting a mailbox will only delete the mailbox, the user will still exist in the database because the user can have access to other mailboxes.

Litigation Hold

Prevents individual mailboxes from being deleted from the archive.

Admin --and users if given rights-- may place individual items under litigation hold in the Search Messages interface.

How a Job Runs

Job Begins

14:55:13, 510[ajp-bio-48009-exec-10] [DEBUG] DeletionJob: saving job: DeletionJob(11FBDB6637DAD22B8019CD9F2A448F96)
14:55:14, 103[DeletionJobTask-1-thread-2] [INFO ] DeletionTaskHelper: Manual trigger of deletion job requested. Not checking schedule
14:55:14, 103[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: DeletionTaskExecutor begins
14:55:14, 103[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Running deletion job.
14:55:14, 103[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: Normal deletion mode

14:55:14, 228[DeletionJobTask-1-thread-2] [DEBUG] DJDeletionOperation: Deletion report stored at E:\archive\Deletion3096775348454516762BGOFMDHQYUUKIHGAPJSQGKSY.html

14:55:14, 868[ActiveMQ Session Task] [TRACE] IndexService: DeletionQuery:delivered:[1980-01-01T21:55:14.000Z TO 2016-04-15T20:55:14.000Z] AND ( uuid:74703C80-1256-0000-A473-666733343035 )
14:55:15, 243[DeletionJobTask-1-thread-2] [INFO ] DJDeletionOperation: Result page TotalHits: 29757 500 in result set
14:55:15, 243[DeletionJobTask-1-thread-2] [TRACE] HibernateStringUtil: inSet: 6728,6730,6731,6729,6734,6732,6735,6733,6719,6736,6737,6740,6738,6739,6741,6742,6744,6745,6743,6746,6747,6748,6749,6753,6752,6750,6751,6754,6757,6756,6755,6759,6758,6761,6760,6763,6762,6765,6764,6767,6769,6768,6766,6770,6771,6773,6772,6775,6777,6776,6774,6778,6779,6781,6780,6783,6782,6785,6784,6786,6790,6787,6792,6791,6789,6788,6793,6794,6795,6796,6797,6801,6805,6798,6802,6799,6800,6803,6807,6804,6808,6809,6806,6810,6811,6812,6813,6814,6817,6818,6815,6816,6819,6820,6823,6821,6822,6825,6826,6824,6827,6828,6829,6830,6831,6832,6833,6837,6834,6838,6835,6836,6840,6839,6841,6842,6845,6844,6843,6846,6850,6849,6847,6848,6851,6854,6853,6852,6857,6855,6856,6858,6859,6860,6862,6861,6865,6863,6864,6867,6866,6870,6869,6868,6871,6873,6872,6875,6874,6878,6877,6876,6879,6880,6882,6883,6881,6884,6885,6886,6890,6887,6888,6891,6889,6892,6893,6894,6898,6895,6899,6896,6900,6903,6897,6901,6902,6906,6904,6907,6908,6905,6909,6910,6911,6912,6915,6913,6914,6918,6916,6919,6917,6922,6920,6924,6923,6921,6925,6926,6930,6928,6927,6932,6931,6929,6933,6934,6938,6936,6935,6940,6939,6937,6942,6944,6943,6941,6945,6946,6948,6947,6949,6950,6952,6951,6953,6955,6954,6956,6960,6959,6957,6961,6958,6962,6963,6966,6964,6965,6968,6967,6971,6969,6973,6970,6974,6972,6976,6975,6977,6979,6981,6978,6982,6980,6984,6983,6985,6986,6988,6987,6989,6990,6992,6991,6993,6995,6994,6996,6997,6999,6998,7000,7004,7001,7003,7005,7002,7008,7007,7009,7011,7006,7010,7012,7016,7013,7015,7017,7019,7014,7018,7020,7024,7021,7023,7025,7022,7026,7027,7029,7030,7028,7031,7032,7034,7033,7035,7037,7039,7036,7040,7038,7042,7041,7043,7045,7044,7046,7047,7049,7048,7050,7054,7051,7053,7052,7055,7056,7058,7062,7057,7059,7060,7061,7063,7066,7064,7065,7067,7068,7069,7070,7071,7072,7073,7075,7074,7078,7076,7077,7079,7080,7081,7082,7083,7084,7085,7086,7087,7091,7088,7089,7090,7094,7095,7092,7093,7096,7097,7098,7099,7102,7103,7100,7101,7106,7107,7104,7105,7108,7109,7110,7111,7114,7115,7112,7113,7116,7117,7118,7119,7120,7121,7122,7123,7126,7124,7128,7125,7129,7130,7127,7134,7131,7135,7132,7136,7133,7137,7138,7139,7142,7140,7141,7143,7146,7144,7145,7150,7147,7148,7151,7152,7149,7153,7154,7158,7155,7157,7156,7160,7159,7162,7161,7163,7165,7164,7166,7167,7169,7168,7170,7174,7171,7173,7172,7175,7178,7177,7176,7180,7179,7182,7186,7181,7185,7184,7188,7183,7187,7190,7189,7193,7192,7191,7194,7196,7197,7195,7198,7202,7200,7201,7204,7199,7205,7203,7206,7210,7208,7212,7209,7207,7213,7211,7216,7214,7215,7218,7217,7220,7222,7224,7221,7219,7223,7226,7225
14:55:15, 243[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: List<EMail>: 6728,6730,6731,6729,6734,6732,6735,6733,6719,6736,6737,6740,6738,6739,6741,6742,6744,6745,6743,6746,6747,6748,6749,6753,6752,6750,6751,6754,6757,6756,6755,6759,6758,6761,6760,6763,6762,6765,6764,6767,6769,6768,6766,6770,6771,6773,6772,6775,6777,6776,6774,6778,6779,6781,6780,6783,6782,6785,6784,6786,6790,6787,6792,6791,6789,6788,6793,6794,6795,6796,6797,6801,6805,6798,6802,6799,6800,6803,6807,6804,6808,6809,6806,6810,6811,6812,6813,6814,6817,6818,6815,6816,6819,6820,6823,6821,6822,6825,6826,6824,6827,6828,6829,6830,6831,6832,6833,6837,6834,6838,6835,6836,6840,6839,6841,6842,6845,6844,6843,6846,6850,6849,6847,6848,6851,6854,6853,6852,6857,6855,6856,6858,6859,6860,6862,6861,6865,6863,6864,6867,6866,6870,6869,6868,6871,6873,6872,6875,6874,6878,6877,6876,6879,6880,6882,6883,6881,6884,6885,6886,6890,6887,6888,6891,6889,6892,6893,6894,6898,6895,6899,6896,6900,6903,6897,6901,6902,6906,6904,6907,6908,6905,6909,6910,6911,6912,6915,6913,6914,6918,6916,6919,6917,6922,6920,6924,6923,6921,6925,6926,6930,6928,6927,6932,6931,6929,6933,6934,6938,6936,6935,6940,6939,6937,6942,6944,6943,6941,6945,6946,6948,6947,6949,6950,6952,6951,6953,6955,6954,6956,6960,6959,6957,6961,6958,6962,6963,6966,6964,6965,6968,6967,6971,6969,6973,6970,6974,6972,6976,6975,6977,6979,6981,6978,6982,6980,6984,6983,6985,6986,6988,6987,6989,6990,6992,6991,6993,6995,6994,6996,6997,6999,6998,7000,7004,7001,7003,7005,7002,7008,7007,7009,7011,7006,7010,7012,7016,7013,7015,7017,7019,7014,7018,7020,7024,7021,7023,7025,7022,7026,7027,7029,7030,7028,7031,7032,7034,7033,7035,7037,7039,7036,7040,7038,7042,7041,7043,7045,7044,7046,7047,7049,7048,7050,7054,7051,7053,7052,7055,7056,7058,7062,7057,7059,7060,7061,7063,7066,7064,7065,7067,7068,7069,7070,7071,7072,7073,7075,7074,7078,7076,7077,7079,7080,7081,7082,7083,7084,7085,7086,7087,7091,7088,7089,7090,7094,7095,7092,7093,7096,7097,7098,7099,7102,7103,7100,7101,7106,7107,7104,7105,7108,7109,7110,7111,7114,7115,7112,7113,7116,7117,7118,7119,7120,7121,7122,7123,7126,7124,7128,7125,7129,7130,7127,7134,7131,7135,7132,7136,7133,7137,7138,7139,7142,7140,7141,7143,7146,7144,7145,7150,7147,7148,7151,7152,7149,7153,7154,7158,7155,7157,7156,7160,7159,7162,7161,7163,7165,7164,7166,7167,7169,7168,7170,7174,7171,7173,7172,7175,7178,7177,7176,7180,7179,7182,7186,7181,7185,7184,7188,7183,7187,7190,7189,7193,7192,7191,7194,7196,7197,7195,7198,7202,7200,7201,7204,7199,7205,7203,7206,7210,7208,7212,7209,7207,7213,7211,7216,7214,7215,7218,7217,7220,7222,7224,7221,7219,7223,7226,7225
14:55:15, 259[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: fromIndex=0,toIndex=100
14:55:15, 728[DeletionJobTask-1-thread-2] [INFO ] GooGooStatementCache: Multiply prepared statement! select folder0_.folder_id as folder_i1_12_0_, folder0_.parent_id as parent_i2_12_0_, folder0_.uuid_mapping_id as uuid_map3_12_0_, folder0_.f_naturalFolderId as f_natura4_12_0_, folder0_.f_name as f_name5_12_0_, folder0_.f_subtype as f_subtyp6_12_0_ from t_folder folder0_ where folder0_.folder_id=?
14:55:15, 728[DeletionJobTask-1-thread-2] [INFO ] GooGooStatementCache: Multiply prepared statement! select folder0_.folder_id as folder_i1_12_0_, folder0_.parent_id as parent_i2_12_0_, folder0_.uuid_mapping_id as uuid_map3_12_0_, folder0_.f_naturalFolderId as f_natura4_12_0_, folder0_.f_name as f_name5_12_0_, folder0_.f_subtype as f_subtyp6_12_0_ from t_folder folder0_ where folder0_.folder_id=?
14:55:21, 306[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: fromIndex=100,toIndex=200
14:55:22, 915[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: fromIndex=200,toIndex=300
14:55:25, 150[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: fromIndex=300,toIndex=400
14:55:28, 384[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: fromIndex=400,toIndex=500
14:55:33, 947[DeletionJobTask-1-thread-2] [TRACE] AbstractDeletionTaskExecutor: deletePage: deleting items from index
14:55:34, 009[DeletionJobTask-1-thread-2] [DEBUG] AbstractDeletionTaskExecutor: deleteIndexDocs
14:55:34, 009[DeletionJobTask-1-thread-2] [DEBUG] AbstractDeletionTaskExecutor: deleteIndexDocs, SearchResults.size: 500
14:55:36, 040[DeletionJobTask-1-thread-2] [TRACE] AbstractDeletionTaskExecutor: sleep for 10 seconds then continue...
14:55:46, 040[DeletionJobTask-1-thread-2] [TRACE] DJDeletionOperation: initializeSearch: offset=0
14:55:46, 040[DeletionJobTask-1-thread-2] [INFO ] DJDeletionOperation: On Page #0

15:09:53, 201[DeletionJobTask-1-thread-2] [INFO ] AbstractDeletionTaskExecutor: DeletionTaskExecutor ends

Delete database nodes

15:09:53, 201[DumpsterThread] [DEBUG] DeletionJobFactory: Dumpster Thread beginning
15:09:53, 201[DumpsterThread] [INFO ] DumpsterDiver: Dumpster dive begins
15:09:53, 404[DumpsterThread] [INFO ] DumpsterDiver: Got these many children of dumpster500
15:09:53, 404[DumpsterThread] [TRACE] DumpsterDiver: Delete 1
15:09:53, 467[DumpsterThread] [INFO ] DumpsterDiver: Deleting node 1

Delete files from disk

15:20:43, 142[DumpsterThread] [INFO ] DumpsterDiver: Delete orphaned documents
15:20:43, 142[DumpsterThread] [INFO ] DeleteDocuments: Removing orphaned documents page
15:20:43, 642[DumpsterThread] [TRACE] DeleteDao: Deleting com.gwava.dao.social.Document id=1, hash=1370D9D230A77EEE0B346898EEC3967EE21B5191CD44674FF1C230A29D004CFE

End deletion job

15:38:43, 477[BackgroundDeletionThread] [TRACE] PartitionCache: Def Partition Path: PARTITION:id=0;name=default;path=E:\archive;bt=1465940323;et=0
15:38:43, 493[BackgroundDeletionThread] [DEBUG] BackgroundDeletionThread: Consider DataStoreBlobID [id=F3E2EA1653D88904DE259C3B88B6C8BE3E04668F0B8C64F1E6EA02A2B3C266C9] for deletion...
15:38:43, 493[BackgroundDeletionThread] [TRACE] BackgroundDeletionThread: DataStoreBlobID [id=F3E2EA1653D88904DE259C3B88B6C8BE3E04668F0B8C64F1E6EA02A2B3C266C9] has 1 entries... 
15:38:43, 493[BackgroundDeletionThread] [TRACE] BackgroundDeletionThread: Delete DB Reference (large blob) for 51452 F3E2EA1653D88904DE259C3B88B6C8BE3E04668F0B8C64F1E6EA02A2B3C266C9
15:38:43, 493[BackgroundDeletionThread] [DEBUG] BackgroundDeletionThread: Removing blob from large blob store with hash of F3E2EA1653D88904DE259C3B88B6C8BE3E04668F0B8C64F1E6EA02A2B3C266C9
15:38:43, 493[BackgroundDeletionThread] [TRACE] PartitionCache: Def Partition Path: PARTITION:id=0;name=default;path=E:\archive;bt=1465940323;et=0

---

Return to Retain Training