Latest revision as of 20:24, 22 June 2018

Whereas the database is used when browsing messages in the Retain mailbox, the indexes are used for performing searches on data stored in Retain. If message metadata, message content, and message attachments have been properly indexed, Retain will be able to find it when the Search feature is used. If that content does not get indexed, then it will not be included in the search results. Making sure that all messages are indexed is very important.

[edit] Advanced Information

[edit] What Happens a When Search Is Performed

When performing a search, Retain simply takes the search conditions and criteria you are searching for and converts them into a query. This query is sent to the Indexer, which returns message IDs for the items it finds. Those message IDs are then looked up in the retain database for specific information about the items themselves.

[edit] Determining Indexer Status

There are several ways to tell if the indexer is running:

The first and easiest way is to just go perform a search. If it returns any results, then the indexer is just fine. Not returning results, however, does not necessarily mean that the Indexer is not running or operational. If the search results are empty, go to step 2.

There is a Retain Java utility, indexerStatus.jsp, that will check to see if the indexer is running. After logging into the RetainServer as admin, add to the end of RetainServer "/Util/indexerStatus.jsp" (e.g., http://10.1.9.26/RetainServer/Util/indexerStatus.jsp). This will take you to a page and tell you what the status is of the indexer.

The status of the Indexer is shown where it reads, "Indexer is alive:"

• True = The indexer is running.
• False = The indexer is not running.

It will also display any items that are in the queue waiting to be indexed or that have not been ::indexed. It will only display 10,000 items and will have a plus ("+") sign next to it if there is more. You can turn the Indexer on or off.

3. Indexer logs. In the /opt/beginfinite/retain/tomcat7/logs directory (Apache Software Foundation/tomcat7/logs in Windows) the indexer has its own logs. They will be titled: Indexer.date.log. Looking in the log, can help to determine if there are errors, or if the indexer is turned off. Below is the initialization and startup of the undexer.

10:25:33,611 LuceneIndexingManager - Indexing manager initialization...
10:25:33,952 LuceneIndexingStats - Stats updater launched
10:25:34,690 LuceneIndexingAddition - Create IndexWriter for Lucene: version=LUCENE_35,path=/retaindata/index/, createMode=false,supportPrefixWildcards=true
10:25:35,728 ServerIndexingBroker - Determining backgroundIndexer for engine:lucene
10:25:38,900 LuceneIndexingManager - Created background indexer...
10:25:38,904 IndexingThread - Start index master
10:25:38,908 LuceneIndexingManager - Indexing manager successfully initialized
10:25:39,516 IndexAdminMessageConsumerImpl - Trying to process the operation: INDEX_LAUNCH_STARTUP
10:25:39,517 IndexAdminMessageConsumerImpl - Will initialize lucene
10:25:39,517 IndexAdminMessageConsumerImpl - INIT: Indexing Manager being launched
10:25:39,517 LuceneIndexingManager - Indexing manager initialization...
10:25:39,561 LuceneIndexingManager - Created background indexer...
10:25:39,561 LuceneIndexingManager - Indexing manager successfully initialized

Here is a snippet from the log on what it looks like when messages are indexed:

10:45:30,966 IndexAdminConfigMessageConsumerImpl - Trying to process the operation: INDEX_INCREMENT_STATS
10:45:32,803 AbstractBackgroundIndexer - processIndexingOfList...
10:45:32,809 LuceneDocumentUtil - NEW LuceneDocumentUtil enabled

[edit] What happens if the Indexer is not working or is turned off?

What if the search produces no results after you have adjusted the date view and reset the search?

The first thing to do is check to see if the indexer is turned off. If it says "...alive: false", try turning the indexer on by clicking on the "Try turning indexer on" button. Take note of the number of items unindexed items. You can also view the unindexed items in the Server status page in Retain.

Another method is to restart tomcat. This will shut down any indexer threads and restart them, bringing the indexer back alive.

Looking in the indexer logs can also help to identify if the indexer is not working if you see this in the log:

10:46:34,125 IndexingThread - End index master
10:46:34,626 NRTSingleton - NRTSingleton: Closing NRTManager
10:46:34,630 NRTSingleton - NRTSingleton: Closing NRTManagerReopenThread
10:46:34,633 LuceneIndexingStats - Stats updater going away
10:46:34,633 LuceneIndexingManager - IndexingManager has shut down all resources

[edit] Identifying Items Not Indexed

Refer to KB article, How to View the Number of Items Not Indexed, for instructions on identifying items that have not been indexed.

[edit] Fixing Missing or Corrupt Index Files

Refer to KB article, Fixing Missing Index Files / Indexer Fails to Load, for instructions on checking for missing index files and/or index file corruption.

[edit] Rebuilding Indexes

Refer to KB article, How to Rebuild Indexes, on symptoms of when indexes may need to be rebuilt and how to rebuild them.

[edit] Explanation of f_indexed Field Values in the t_message table

When wanting to see a summary count of messages by each indexing state, run the query described in this kb: SQL query for showing the message count of various index states

[edit] Retain 3.x:

• 1: Fully indexed
• -1: Indexing error. No part of this message was indexed.
• -32: Larger than indexing size limit.
• -64: Retain doesn't recognize this file type. Therefore, it doesn't know how to extract text from it to index. (I.E. attachment/item was not indexed)
• -128: INDEXING_HIBERNATE_EXCEPTION It's unlikely that you'll ever see this code. From Development: I've looked through the code and can't find any place where a hibernate exception is thrown, however, tracking down where exceptions can come from after the fact can be difficult so it is possible that I just missed it. However, if a hibernate exception is thrown this error code means that during the process of indexing this message that an hibernate exception occurred that prevented the indexing of this message.

[edit] Retain 4.x:

• 2: Fully indexed
• 10: Indexing Error (same as -1 in Retain 3.x)
• 16: General indexing error (This should never be written to the database.)
• 32: Larger than indexing size limit. (Same as -32 on 3.x)
• 64: INDEXING_NOT_ON_WHITELIST (Same as -64 on 3.x)
• 128: INDEXING_HIBERNATE_EXCEPTION (Same as -128 on 3.x, this should never be written to the database.)
• 256: INDEXING_EXTRACTOR_EXCEPTION. As per Dev: Any text extraction we perform on any file type could potentially throw an exception and this is the f_indexed value that we set to indicate this. (This should never be written to the database.)
• 512: INDEXER_EXCEPTION. (As of 2015-02-10 not actually being used in the beta code.)

Note: Just because it shouldn't be written to the database doesn't mean you won't ever see it. Typically you'll want to assume that anything greater than a 2 had an error and is either partially indexed or not indexed at all.

[edit] Tweaking Indexing speed and other tools

In the file /opt/beginfinite/retain/RetainServer/WEB-INF/classes/config/lucene.indexing.properties There are a number of items that can be tweaked.

[edit] All Things SOLR

[edit] Solr MemoryMap Files and Memory Usage

Here is a blog post from someone that works for Solr about its use of MemoryMap files and memory usage in general: Understanding Solr Memory Usage

[edit] Accessing the Solr Web Interface

To access the HPI web interface, go to http://[dns hostname / IP]:8081/hpi/#/

NOTE: The username and password is stored in the ASConfig.cfg within the dynamicAttributes tag:

    
     <dynamicAttributes>
        <entry>
          <string>hpiPassword</string>
          <string>retain</string>
        </entry>
        <entry>
          <string>hpiUsername</string>
          <string>admin</string>
        </entry>
      </dynamicAttributes>

This interface can be used to test queries. Simply to go "retaincore" and then to Query. In the second field from the top under a section header "common", you specify the field you are searching then the search criteria. For example, if I'm looking for a particular message ID of "187234", I would write the query like this: id:187234 and then click on the blue Execute Query button. In the window to the right of the query dialog, it will display some data:

  "responseHeader": {
    "status": 0,
    "QTime": 371,
    "params": {
      "q": "id:2",
      "indent": "true",
      "wt": "json",
      "_": "1442356979581"
    }
  },
  "response": {
    "numFound": 1,
    "start": 0,
    "docs": []
  },
  "facet_counts": {
    "facet_queries": {},
    "facet_fields": {},
    "facet_dates": {},
    "facet_ranges": {},
    "facet_intervals": {},
    "facet_heatmaps": {}
  }
}

Look under "response" at the "numFound":. The number displayed is the number of hits it received. In this example, it found that a message with the ID of 187234 had a hit, which means that message was indexed.

[edit] Max Docs vs. Num Docs

numDocs represents the number of searchable documents in the index. maxDoc may be larger as the maxDoc count includes logically deleted documents that have not yet been removed from the index. (From Solr official Tutorial). You could remove logically deleted files by optimizing your index.

[edit] How SOLR Determines the Relevancy Score

• SOLR Relevancy FAQ
• Very Technical Data on SOLR Relevancy

From product management:

I did some research too and this subject is not an easy one. Everyone has a preference. Dania Bilal did a research article on "Ranking, relevance judgment, and precision of information retrieval on children's queries: Evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids" which was published and available for hire or purchase here: http://onlinelibrary.wiley.com/doi/10.1002/asi.22675/abstract

The winner was Google by the way... 11% overlap vs 3% for Yahoo kids...

I think we release what we have and get feedback from our customers.

[edit] Retain 4.0 Demo

Demo server credentials that can be used to demonstrate the latest version:

http://retainbeta.gwava.com/RetainServer 
admin / myretain4beta

[edit] 4.x Indexer Tweaks

You can override the default thread allocation mechanism in .../RetainServer/WEB-INF/classes/config/solrcloud.indexing.properties file.

[edit] Retain 4.0 Training Video: New Search UI / External HPI Installation and Configuration

• Retain 4.0 Training, delivered on August 28, 2015 by Matt Southwick (MP4)

• Retain 4.0 Training, delivered on August 28, 2015 by Matt Southwick (Requires TeamViewer)

[edit] HPI - External Indexer (soon to be it's own page)

Default file locations

[edit] Windows

Index Manager:

C:\Program Files (x86)\GWAVA\Retain

Tomcat:

C:\Program Files\Retain\Tomcat7

Shard:

C:\Program Files (x86)\GWAVA\Retain\HPI

[edit] Linux

Index Manager:

/opt/beginfinite/retain/idxmanager

Tomcat:

/opt/beginfinite/retain/hpi-tomcat

Shard:

/media/retain-hpidata

[edit] Index Schema-Version

Q. Has someone a short description what the Index schema-version tells us in retain/ is good for? I've seen servers on 4.1.0.1 with schema version 403 and some on 4.1.0.1 with 340, is this okay, or should i be concerned that there might be something wrong?!?

A. There was a thought to change the indexing schema but the cost to the customers was too high so it was taking out for now. All that number tells you is what the indexing schema is, when the change is made and after the migration that number will change. For now it does nothing.

For the index schema version: The only difference between 403 and 340 is additional Chinese (either that or some of the other Asian languages) character support. Other than that, there's nothing different.

NOTE: if they're on 340 then they installed 4.x on either 4.0, 4.0.1, or 4.0.2. The "403" schema denotes 4.0.3.x and later.

[edit] Solr memory issue

There’s a tiny little issue with solr and tomcat in Retain 4.5 and below.

Tomcat is not giving solr enough time to shutdown properly when we tell tomcat to stop, so it knocks the feet out from under solr since solr doesn’t have enough time to shut itself down properly. This causes orphaned files and memory leaks.

We need to stop tomcat and then make sure the index .lock file is gone before starting tomcat again. If tomcat is stopped and the .lock file still exists it can be safely deleted.

Windows

…\index\solrhome\retaincore\data\index\write.lock

Linux

.../index/solrhome/retaincore/data/index/write.lock

For example:

/var/opt/beginfinite/retain/index/solrhome/retaincore/data/index/write.lock

If memory is not clearing a reboot of the server will be needed. Stop tomcat, remove the write.lock file, then restart the server.

Fix is set to be going into Retain 4.6

[edit] Go to Server Configuration