Search plugins¶

The search service determines the type of information that is indexed and searchable via a plugin mechanism.

See Searchlight Plugin Documentation for plugin installation and general configuration information.

See each plugin below for detailed information about specific plugins:

Indexing model¶

The Mitaka Searchlight release introduced the ability to continue executing search requests while reindexing operations are running. This feature is called zero-downtime reindexing. In order to implement zero-downtime indexing, the concept of a resource group was introduced.

A resource group is a collection of plugins that share an Elasticsearch index. Since each plugin represents a resource type, you can think of a resource group as a collection of resource types.

For each resource group, Searchlight creates an index whose name consists of the resource group name appended with a timestamp. Each resource group is referred to by a pair of Elasticsearch aliases. One alias is used for searching by the API (the search alias), and the other (the listener alias) is used to index incoming events.

During reindexing, a new index is created, and the listener alias is pointed at both the old and new indices. Incoming events are therefore indexed into both indices. The search alias is left pointing at the old index. Once indexing is finished, both aliases are pointed solely at the new index and the old index is deleted.

In order to improve the performance of reindexing, index refresh of the new index is disabled during reindexing, and turned on after reindexing is done. As a consequence, Documents synced to the new index are not searchable until index is refreshed, but document retrieval by IDs still works, because GET operation in Elasticsearch is realtime.

It is important to note that zero-downtime reindexing requires that all plugins in a resource group are indexed together. When it’s desired to index an individual resource type, an optimization copies existing data directly from the old index to the new one to avoid re-harvesting the data from each service API.

Bulk indexing¶

To initially create the catalog index (or add new resource typs to it later), run the following command:

$ searchlight-manage index sync

This will iterate through all registered and enabled search plugins and request that they perform a full indexing of all data that’s available to them.

It is also possible to index just a single resource, or all resources belonging to a resource group. For instance, to index all glance images:

$ searchlight-manage index sync --type OS::Glance::Image

As described above, this will create a new index for all plugins that share a resource group with OS::Glance::Image. The management command will retrieve up-to-date information from the Glance API. Data for other plugins will be bulk-copied from a preexisting index into the new one using the scroll and bulk features of Elasticsearch.

You can use the wildcard character * at the end of the type argument. For instance, the following will match all cinder plugins:

$ searchlight-manage index sync --type OS::Cinder::*

Wildcard characters are only allowed at the end of the argument; they will not be matched anywhere else.

To index all resources in the ‘searchlight’ resource group:

$ searchlight-manage index sync --index searchlight

You will be prompted to confirm unless --force is provided.

To reindex resources only without notification:

$ searchlight-manage index sync --notification-less

The searchlight-manage index sync command may be re-run at any time to perform a full re-index of the data. As described above, there should be no or very little impact on search requests during this process.

Parent/child relationships¶

Some plugins contain multiple resources with parent/child relationships; the Designate plugins are an example. Because reindexing parent data independent of child documents does not logically make sense (without orphaning them), it is not possible to request indexing of a child resource type:

$ searchlight-manage index sync --type OS::Designate::RecordSet

'OS::Designate::RecordSet' is a child of 'OS::Designate::Zone' and cannot be indexed separately.
Indexing 'OS::Designate::Zone' will re-index all child resource types.

You can see parent/child relationships in the list of resources presented prior to indexing:

$ searchlight-manage index sync --type OS::Designate::Zone

Resource types (and indices) matching selection:
  OS::Designate::Zone (designate)
      ---->  OS::Designate::RecordSet

Child plugins will inherit their resource group name from their parent. Any child configuration setting for resource_group_name will be ignored.

Incremental Updates¶

Once a resource has been indexed, typically you will only need to consume incremental updates rather than re-index the entire data set again. The preferred methodolgy is to set up notification listening.

$ searchlight-listener

Multi-Thread Support¶

The Newton Searchlight release introduced multiple thread support for indexing. Previously when the searchlight-manage index sync command was executed, all indexing occurred in a single thread. To boost the performance of the indexing functionality, each resource type will now index in its own thread. Multiple indexing threads will run concurrently.

By default, the maximum number of simultaneous threads is 3. This limit can be modified in the Searchlight configuration file. The setting is called workers and lives under [manage]. For example, to increase the maximum number of threads to 6, the following can be added to the Searchlight configuration file:

[manage]
workers=6

The use of threads can also affect the parsing of the log files. The default formatting of the log messages include only the process ID, but no thread-specific information. This can be changed by modifying the formatting string settings in the Searchlight configuration file. To add the thread ID for a message, add %(thread)d. To add the thread name, add %(threadName)s. For example, to add the thread ID and the thread name after the process ID to the logging message, the following setting can be added to the Searchlight configuration file:

logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(thread)d %(threadName)s %(levelname)s %(name)s [-] %(instance)s%(message)s

Force Elasticsearch indexing¶

The Newton Searchlight release introduced the ability to reindex from Elasticsearch only, bypassing the plugin APIs altogether. This option is useful if there has been a change to the mapping definitions or the index settings. This functionality is enabled with the option --apply-mapping-changes for the index command.

A sample usage would be:

$ searchlight-manage index aliases --apply-mapping-changes

The --type option is not compatible with the --apply-mapping-changes option. Specifying both options on the command line will result in an error.