TeamWox SDK: Search and Filtering - Part 1

 

Introduction

TeamWox groupware initially offers effective and convenient search over modules from standard delivery. But how to handle data from custom modules? The answer is obvious - you need to enhance your module functionality with searching feature using TeamWox API.

The Search module enables data searching in all TeamWox modules. When you search for data, user access permissions are taken into account. In this article we will consider working with the Search module's API. Using the Hello World module as an example, you will learn how to organize data indexing and searching for custom module.

  How the 'Search' Module Works
  'Search' Module API
  Example of Implementing Search for a Custom Module
    1. Including 'Search' Module API into a Project
    2. Implementing the ISearchClient Interface
    3. Extending Module's Manager
    4. Indexing
        4a. Indexing a Single Record
        4b. Full Re-indexing of a Module
    5. Display Search Query Results
    6. Demonstration

 

How the 'Search' Module Works

Searching information in TeamWox groupware is performed using the Search module, that comes in standard delivery. This module equally works with all other TeamWox modules - first it indexes and processes their data and then users are able to search information on a separate page by entering search queries.

According to the TeamWox ideology, to extend module's functionality you must implement an interface from TeamWox API. As for search, in custom module we must implement the ISearchClient interface provided by search module API.

Consider the principle of the 'Search' module on the following scheme:

The Principle of the 'Search' module

Next, for simplicity we will imply the search module as Search and custom module or module from standard delivery as Module.

There are two interaction modes of Search and Module.

1. Indexing Data Real-Time. Module applies to Search to inform of changed data.

2. Full Re-indexing. Search applies to a certain Module in case of full re-indexing that is carried out either by schedule or by request of TeamWox administrator. In this case Module provides all available information about its data to Search.

Module's developer decides how to store data (in DBMS or in file storage) and what data to provide Search for indexing and display.

Display Search Results

Search stores only identifiers of Module's records (+ other service information) in a special storage (see also the How to Speed Up TeamWox - Store Components on Different Drives article). To display search query results, Search applies to Module to fill information about found record. Record id is passed as an argument.

 

Search' Module API

Search' Module API (this module ID is TWX_SEARCH) is included into TeamWox SDK. You can find it in the "<TeamWox SDK_install_dir>\SDK\API\Search.h" file. Consider API interfaces:

  • ISearchManager - the main interface used to invoke Search functionality.
  • ISearchIndexer - interface of indexing data.
  • ISearchClient - Module must implement this interface to interact with Search.
  • ISearchResult - interface to display search results. This object is passed into module to fill data. "Set" methods are used to fill object with data and "Get" methods - to display data on page with search results.
  • ISearchRequest - interface of requesting data with custom access rights.

You can find the complete description of the 'Search' module in TeamWox SDK documentation.

 

Example of Implementing Search for a Custom Module

Consider the implementation of custom module search on example of the Hello World module. As a starting point you can download and unzip the archive for the previous publication Setting Up Custom Modules Environment - Part 2 from TeamWox SDK articles series. The final variant with all modifications described is available in attachment to this article.

Currently, for the Hello World module we have implemented the following features: adding records and storing them in DBMS and file storage, and also creating reports for these data. The next step of extending functionality - ability to index module's data for their searching, and also maintaining search index in up-to-date state when modifying or deleting data.

Our implementation will be based on the Board module source code, that is available in TeamWox SDK.

 

1. Including 'Search' Module API into a Project

1. Open the "HelloWorld.sln" project, then in Visual Studio open the "stdafx.h" file.

2. Include the "Search.h" file. In our example module project is located in the "<TeamWox SDK_install_dir>\Modules\HelloWorld\" folder, so we will use the relative path up two levels. Correct it if you have unpacked the project into a different folder.

//---
#include "..\..\SDK\API\Core.h"
#include "..\..\SDK\API\Search.h"
//---

3. Add the "<TeamWox SDK_install_dir>\SDK\API\Search.h" file to the project by placing it to the "\Header Files\API\" folder.

Including 'Search' Moduel API to the Hello World Project

 

2. Implementing the ISearchClient Interface

1. In the project add new class for the search client by placing it in managers category (according to accepted hierarchy).

Declaring Class of the Search Client              Implementing Class of the Search Client

2. Declare the CHelloWorldSearch class that implements methods of the ISearchClient interface. We will implement the ISearchClient::Release method right here in class declaration.

//+------------------------------------------------------------------+
//|                                                          TeamWox |
//|                   Copyright 2006-2010, MetaQuotes Software Corp. |
//|                                        https://www.metaquotes.net |
//+------------------------------------------------------------------+
#pragma once

//+------------------------------------------------------------------+
//| Class for the search client                                      |
//+------------------------------------------------------------------+
class CHelloWorldSearch : public ISearchClient
  {
private:
   IServer          *m_server;            // TeamWox server's interface
   class CHelloWorldManager &m_manager;   // our internal interface

public:
                     CHelloWorldSearch(IServer *server,class CHelloWorldManager &manager);
                    ~CHelloWorldSearch();

   //--- Interface methods
   void              Release() { delete this; }
   TWRESULT          Reindex(ISearchManager *search_mngr);
   TWRESULT          FillResult(const Context *context, ISearchResult *result);
   TWRESULT          ModifyRequest(const Context *context, ISearchRequest *request);

private:
   void              operator=(CHelloWorldSearch&) {}
  };
//+------------------------------------------------------------------+

3. Specify the ISearchClient interface (and the CHelloWorldSearch class that implements it) in the list Hello World module's implemented interfaces.

//+------------------------------------------------------------------+
//| Get module's interfaces                                          |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldModule::GetInterface(const wchar_t *name, void **iface)
  {
//--- Check
   if(name==NULL || iface==NULL) return(RES_E_INVALID_ARGS);
//---
   if(StringCompareExactly(L"IToolbar",name))               
     { *iface=&m_toolbar;                                 return(RES_S_OK); }
   if(StringCompareExactly(L"IModuleMainframeTopBar",name)) 
     { *iface=static_cast<IModuleMainframeTopBar*>(this); return(RES_S_OK); }
   if(StringCompareExactly(L"IModuleTips", name))           
     { *iface=static_cast<IModuleTips*>(this);            return(RES_S_OK); }
   if(StringCompareExactly(L"IWidgets",name))               
     { *iface=&m_widgets;                                 return(RES_S_OK); }
   if(StringCompareExactly(L"ISearchClient",name))
     {
      *iface=new (std::nothrow) CHelloWorldSearch(m_server,m_manager);
      if(*iface==NULL) ReturnError(RES_E_OUT_OF_MEMORY);
      return(RES_S_OK);
     }
//---
   return(RES_E_NOT_FOUND);
  }

4. Re-indexing and displaying search results in fact will be implemented in manager's module, while in the ISearchClient::Reindex and ISearchClient::FillResult methods we will simply pass control into the appropriate methods of module's manager.

  • ISearchClient::Reindex
//+------------------------------------------------------------------+
//| Full Re-indexing of Module                                        |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldSearch::Reindex(ISearchManager *search_manager)
  {
//--- Start re-indexing in manager
   return(m_manager.SearchReindex(search_manager));
  }
  • ISearchClient::FillResult
//+------------------------------------------------------------------+
//| Fill search result                                               |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldSearch::FillResult(const Context *context, ISearchResult *result)
  {
//--- 
   return(m_manager.SearchFillResult(context,result));
  }

5. In our example there is no need to implement request with custom access rights.

//+------------------------------------------------------------------+
//| Modify requests                                                  |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldSearch::ModifyRequest(const Context* /*context*/, ISearchRequest* /*request*/)
  {
   return(RES_E_NOT_IMPLEMENTED);
  }

 

3. Extending Module's Manager

In module's manager we have to implement previously mentioned methods of re-indexing records (SearchReindex) and displaying search results (SearchFillResult). For this we have to get reference to the ISearchManager interface.

On updating record (the CHelloWorldManager::Update method) we will invoke procedure of its indexing (the SearchReindexRecord method), and on deleting record (the CHelloWorldManager::InfoDelete method) - we will delete record information and its comments from the search index (the SearchRemoveRecord method).

1. In class of module's manager declare pointer to the search manager interface.

//+------------------------------------------------------------------+
//| Module's manager                                                 |
//+------------------------------------------------------------------+
class CHelloWorldManager : public IReportsProvider
  {
private:
   static HelloWorldRecord m_info_records[];           // List of public information
   static HelloWorldRecordAdv m_advanced_records[];    // List of information with limited access
   //--- SET THE VALUE ONLY WHEN INITIALIZING
   IServer          *m_server;                         // Reference to server
   IFilesManager    *m_files_manager;                  // files manager
   ISearchManager   *m_search_manager;                 // 'Search' module's manager
   IComments        *m_comments_manager;               // Comments manager

2. As the Search is a separate NON SYSTEM module, we must get its interface (and interfaces of all other NON SYSTEM modules) on the second step of initialization.

  • Description
//+------------------------------------------------------------------+
//| Module's manager                                                 |
//+------------------------------------------------------------------+
class CHelloWorldManager : public IReportsProvider
  {
public:
                     CHelloWorldManager();
                    ~CHelloWorldManager();
   //---
   TWRESULT          Initialize(IServer *server, int prev_build);
   TWRESULT          PostInitialize();
  • Implementation
//+------------------------------------------------------------------+
//| Second step of manager initialization                              |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::PostInitialize()
  {
//--- Checks
   if(m_server==NULL) ReturnError(RES_E_INVALID_ARGS);
//--- Get reference to interface of the search. In case of failure just continue to work (don't inform of it)
   m_server->GetInterface(TWX_SEARCH,L"ISearchManager",(void**)&m_search_manager);
//---
   return(RES_S_OK);
  }

3. In module's manager declare the following methods:

  • SearchReindexRecord - Indexes records when it is changed
  • SearchRemoveRecord - Deletes record from search index
  • SearchFillResult - Generates page with search results
  • SearchReindex - Performs full re-indexing of module's data
  • ReadDescription - Utility method of reading record's contents from file storage
  • SearchRecordAdd - Indexes contents read from file storage
//+------------------------------------------------------------------+
//| Module's manager                                                 |
//+------------------------------------------------------------------+
class CHelloWorldManager : public IReportsProvider
  {
  
public:
   //--- Search
   TWRESULT          SearchReindex(ISearchManager *search_manager);
   TWRESULT          SearchFillResult(const Context *context,ISearchResult *result);
private:
   //---
   TWRESULT          ReadDescription(HelloWorldRecord* record,
                                     wchar_t**         description,
                                     size_t*           description_len,
                                     size_t*           description_alocated);
   //---
   TWRESULT          SearchRecordAdd(ISearchManager*   search_manager,
                                     ISearchIndexer*   indexer,
                                     HelloWorldRecord* record,
                                     const wchar_t*    description,
                                     size_t            description_len);
   //---
   TWRESULT          SearchReindexRecord(const Context* context,HelloWorldRecord* record);
   TWRESULT          SearchRemoveRecord(const Context* context,INT64 record_id);

4. Add the call of re-indexing procedure to the CHelloWorldManager::Update method. So that on any change of record its information indexing is guaranteed.

//+------------------------------------------------------------------+
//| Add and save new record in HELLOWORLD table                      |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::Update(const Context *context,HelloWorldRecord *record,AttachmentsModify *attachments)
  {
.............................
//--- Re-index the message
   SearchReindexRecord(context,record);
//---
   return(RES_S_OK);
  }

5. Add the call of procedure of deleting record information and its comments from search index to the CHelloWorldManager::InfoDelete method.

//+------------------------------------------------------------------+
//| Delete record from HELLOWORLD table                              |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::InfoDelete(const Context *context,INT64 id)
  {
.............................
//--- Remove record from search index
   SearchRemoveRecord(context,id);
//---
   return(RES_S_OK);
  }

 

4. Indexing

In TeamWox groupware indexing is performed with a certain delay that depends on frequency of adding new content and its amount. This approach optimizes the load on server. First data are accumulated, and then (when certain amount is accumulated or after some time) they are processed in batch mode.

It is more efficiently to process larger amount of data, than one record at a time.

In the HelloWorldManager.h file create the EnHelloWorldSearchId enumeration containing two IDs. These IDs will determine categories of indexed records: first - the contents of record, second - contents of comments.

enum EnHelloWorldSearchId
  {
   HELLOWORLD_BODY_SEARCH_ID   =0,
   HELLOWORLD_COMMENT_SEARCH_ID=1
  };

 

4a. Indexing a Single Record

1. First create pointer to the ISearchIndexer interface. Every time when re-indexing starts, new indexer object is created. It accumulates all data related to a record.

//--- Get indexer's object
   CSmartInterface<ISearchIndexer> indexer;
   if(RES_FAILED(res=m_search_manager->GetSearchIndexer(&indexer)

2. Record's contents are read and then after processing they are added to the indexer.

//--- Read description
   ReadDescription(record,&description,&description_len,&description_allocated);
//--- Add/Update record in search index
   SearchRecordAdd(m_search_manager,indexer,record,description,description_len);

3. Comments is the standard TeamWox feature. Developers don't need to create procedures of indexing their contents. All the necessary tools are provided by the IComments interface. Particularly, indexing comments is performed using the IComments::SearchReindexRecord method.

//--- Indexing comments
        m_comments_manager->SearchReindexRecord(context->sql,m_search_manager,indexer,0,NULL,0,
                                                HELLOWORLD_COMMENT_TYPE_COMMENTS,0,record->id,
                                                HELLOWORLD_COMMENT_SEARCH_ID);

4. Key steps in implementing the SearchRecordAdd method.

  • First we will put record's data into the indexer's object. In our case it is plain text without HTML formatting tags.

    In our example we will give more weight to titles (2.0) and less weight to description (1.0). So that record with keywords in its title will be more relevant than record with the same keywords only in description.
//---
   indexer->AddText(SEARCH_PLAIN_TEXT,record->name,2.0f);
//---
   if(description!=NULL && description_len>0)
      indexer->AddText(SEARCH_PLAIN_TEXT,description,description_len,1.0f);
  • After that we have to juxtapose these data with some unique id using the ISearchManager::Insert method. In our example it will consist of two parts: the HELLOWORLD_BODY_SEARCH_ID constant (from the EnHelloWorldSearchId enumeration we have created) and record's id in DBMS.
//--- Add indexed data
   search_manager->Insert(HELLOWORLD_MODULE_ID,SEARCH_GENERIC,HELLOWORLD_BODY_SEARCH_ID,record->id,indexer);

In general, it is enough to specify only one of the record1 or record2 arguments in the ISearchManager::Insert method. Ability to specify second id makes implementation of search more flexible.

Choosing the record1 and record2 parameters for each given implementation of indexing is up to you as a developer.

5. When record's contents are deleted, the search index is purged from record contents and its comments' contents.

//+------------------------------------------------------------------+
//| Delete record from search index                                  |
//+------------------------------------------------------------------+
TWRESULT CHelloWorldManager::SearchRemoveRecord(const Context *context,INT64 record_id)
  {
//--- Checks
   if(context==NULL || m_search_manager==NULL || m_comments_manager==NULL) ReturnError(RES_E_INVALID_ARGS);
//---
   m_search_manager->Remove(HELLOWORLD_MODULE_ID,SEARCH_GENERIC,HELLOWORLD_BODY_SEARCH_ID,record_id);
//---
   m_comments_manager->SearchRemove(context,m_search_manager,HELLOWORLD_COMMENT_TYPE_COMMENTS,record_id,HELLOWORLD_COMMENT_SEARCH_ID);
//---
   return(RES_S_OK);
  }

4b. Full Re-indexing of a Module

This procedure is similar to indexing a single record. The Search module clears the search index and sequentially indexes all records from selected modules. The difference is in the range of use - in some cases it is more efficiently to perform full re-indexing rather than to index each record:

  • Emergency stop of server. In this case full re-indexing is launched automatically.
  • Emergency stop of server while performing full re-indexing. In this case once started procedure must be completed.
  • Errors in a module.
  • Massive deleting of records.

Full re-indexing is performed in the service time according to schedule (once a week). For this reason search indices are not included into backup.

Consider implementing the SearchReindex method of full re-indexing.

1. In one query select all records.

//---
   CSmartSql sql(m_server);
   if(sql==NULL) ReturnError(RES_E_FAIL);
//--- Text of SQL request to get row from HELLOWORLD table by specified ID.
   char query_select_string[]="SELECT id,name,category,description_id,attachments_id FROM helloworld";
//--- "Bind" data to parameters of request
   SqlParam params_query_select_string[] ={
      SQL_INT64, &record.id,            sizeof(record.id),
      SQL_WTEXT,  record.name,          sizeof(record.name),
      SQL_LONG,  &record.category,      sizeof(record.category),
      SQL_INT64, &record.description_id,sizeof(record.description_id),
      SQL_TEXT,   record.attachments_id,sizeof(record.attachments_id)
      };
//--- Send request
   if(!sql.Query(query_select_string,
                  NULL, 0,
                  params_query_select_string, _countof(params_query_select_string)))
      ReturnErrorExt(RES_E_SQL_ERROR,NULL,"helloworld record query failed");

2. Perform indexing of all records one by one. Just like for a single record, we perform indexing of all records and their comments.

//--- Get response
   while(sql->QueryFetch())
     {
          description_len=0;
      //--- Read description
          ReadDescription(&record,&description,&description_len,&description_allocated);
      //--- Add/Update record in search index
          SearchRecordAdd(m_search_manager,indexer,&record,description,description_len);
      //--- Indexing comments
          m_comments_manager->SearchReindexRecord(sql.Interface(),
                                                  search_manager,
                                                  indexer,0,NULL,0,
                                                  HELLOWORLD_COMMENT_TYPE_COMMENTS,
                                                  0,record.id,
                                                  HELLOWORLD_COMMENT_SEARCH_ID);

3. It is recommended to display debug messages about current state of re-indexing process (for example, after every 500 successfully indexed records). At the end display the total number of indexed records.

      //---
      if((++count)%500==0)
         ExtLogger(NULL, LOG_STATUS_INFO) << count << " records re-indexed";
      //--- Clean-up
      ZeroMemory(&record,sizeof(record));
     }
//---
   ExtLogger(NULL, LOG_STATUS_INFO) << count << " records re-indexed";
//---
   if(description!=NULL) { delete [] description; /*description=NULL;*/ }
//--- Release request
   sql.QueryFree();
//---
   return(RES_S_OK);
There may be situations when search index still stores record's information in DBMS and file storage, but the data is either deleted or inaccessible. Full re-indexing (performed by administrators request or according to schedule) completely solves this problem.

 

5. Display Search Query Results

Search query returns only IDs of found records. Module's developer decides what part of record's contents to display using these IDs.

Consider implementing the ISearchClient::FillResult method (in the CHelloWorldManager::SearchFillResult method).

1. Get IDs of found records.

//--- Check parameters
   const SearchResult *data=result->GetResult();
   if(data==NULL || data->module_id!=HELLOWORLD_MODULE_ID) 
      ReturnErrorExt(RES_E_INVALID_ARGS,context,"invalid parameters for search result");

2. We have to determine, what type of records the found id is referring to - either to record itself, or to its comments.

//--- Determine where to get information from
   switch(data->record1)
     {
      case HELLOWORLD_COMMENT_SEARCH_ID:
         m_comments_manager->SearchFillResult(context,result,data->record2,comment_url,_countof(url),&record_id);
         break;
      case HELLOWORLD_BODY_SEARCH_ID:
      default:
         record_id=data->record2;
     }

In case of comments all the necessary tools are provided by the IComments interface. Particularly, displaying search results in comments is performed using the IComments::SearchFillResult method.

You have probably noticed, that often several pages with the same title are displayed on the search results page. This means that phrase you were searching for is found in several comments at once. Every such comment is displayed on a separate page.

3. Get record from DBMS by found id.

//---
   if(record_id==0) return(RES_E_NOT_FOUND);
//---
   if(RES_FAILED(res=Get(context,record_id,&record)))
     {
      ExtLogger(context,LOG_STATUS_ERROR) << "failed to get record info #" << record_id << " for search [" << res << "]";
      return(res);
     }

4. Now we have to fill page with search results. In general, such a page consists of record title (as hyperlink), brief description (includes part of found record's contents), information about modification date, list of assigned workers and so forth. In our example we will limit ourselves with title and description. Modification date is displayed in search result automatically.

  • Title
//--- Set the title
   result->SetTitle(record.name);
  • Link for title
//--- Generate URL
   if(comment_url[0]!=0)
      StringCchPrintf(url,_countof(url),L"/helloworld/number_two/view/%I64d?%s",record_id,comment_url);
   else
      StringCchPrintf(url,_countof(url),L"/helloworld/number_two/view/%I64d",record_id);
//---
   result->SetUrl(url);
  • Description
//--- Generate description
   if(data->record1==HELLOWORLD_BODY_SEARCH_ID && record.description_id>0)
     {
      if(RES_SUCCEEDED(m_files_manager->FileInfoGet(NULL,record.description_id,&file_info)) && 
         file_info.size>0 && (file_info.size%2)==0)
         {
         wchar_t description_stat[512]={0};
         //---
         if(file_info.size>sizeof(description_stat))
            {
            UINT64   len        =file_info.size;
            wchar_t *description=new wchar_t[size_t(len/sizeof(wchar_t))];
            //---
            if(description==NULL) 
               ReturnErrorExt(RES_E_OUT_OF_MEMORY,context,"failed not enough memory for description content");
            //---
            if(RES_SUCCEEDED(m_files_manager->FileRead(NULL,record.description_id,description,&len,0)))
               result->AddText(SEARCH_PLAIN_TEXT,description,int(len/sizeof(wchar_t)));
            //---
            delete []description;
            }
         else
            {
            UINT64 len=file_info.size;
            if(RES_SUCCEEDED(m_files_manager->FileRead(NULL,record.description_id,description_stat,&len,0)))
               result->AddText(SEARCH_PLAIN_TEXT,description_stat,int(len/sizeof(wchar_t)));
            }
         }
      }

 

6. Demonstration

Let's demonstrate implemented features.

1. On the Page 2 add new record in WYSIWYG editor.

Adding Record for Indexing

2. Wait a while of perform full re-indexing of the Hello World module.

3. Search previously added data.

Searching Previously Added Data

4. Search results are displayed on a separate page.

Search Results

 

Conclusion

As you see, it is not difficult to implement the search functionality in your module. In the second part we will talk about filtering search results.


helloworld-search-part1-en.zip (339.42 KB)

2011.09.07