Document version: 245. Automatically generated.

📘

Elasticsearch is a search engine based on the Lucene library.

It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

ElasticSearch 6.8.23 is used as a storage for transcriptions of all conversations conducted with bots.

🚧

ElasticSearch installation is customized

Our installation includes several third party ES plugins which are mandatory for successful deployment.
Hence it's not possible to use vanilla ES instance!

Prerequisites

• Access to GIT repository repo-ext.sentione.com has been granted
• Access to hub.sentione.com RPM repository has been granted
• Dedicated Automate configuration repository has been cloned to local filesystem (eg. /home/user/automate-config)
• 2 dedicated machines with Red-Hat based operating system (eg. Red-Hat Enterprise, CentOS Stream 9) available for installation

Conventions

• CAPITAL_LETTERS_VARIABLE_NAME - fragment which should be replaced with actual values matching particular environment
• AUTOMATE_CONFIG_REPO_PATH - Path to automate configuration repository (eg. /home/user/automate-config)
• ENVIRONMENT_NAME - Environment name (eg. production)

Example

$ cd AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services

Should be substituted with:

$ cd /home/user/automate-config/production/external-services

Steps

1. Filling in configuration template

You should start with filling in ElasticSearch placeholders found in file
AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/inventory

(eg. IP addresses like x.x.x.x).

1.1 RPM repository credentials

📘

SentiOne should provide credentials to RPM repository at https://hub.sentione.com

Please put these credentials in following section

[all:vars]
NEXUS3URL=https://hub.sentione.com
NEXUS3USER=PUT_YOUR_NEXUS_USERNAME_HERE
NEXUS3PASSWORD=PUT_YOUR_NEXUS_PASSWORD_HERE

1.2 Cluster configuration

[elasticsearch:vars]
ELASTICSEARCH_USER=es_user_name
ELASTICSEARCH_PASSWORD=es_password
ELASTICSEARCH_SECURITY_CA_PASSWORD=ca-certificate-password-here

ELASTICSEARCH_HTTP_PORT=9200
ELASTICSEARCH_TRANSPORT_PORT=9300
ELASTICSEARCH_CLUSTER_NAME=example-cluster-name

1.3 Master node configuration

[elasticsearch-primary]
x.x.x.x ansible_user=root
...
[elasticsearch:vars]
NODE_IP=x.x.x.x
NODE_DNS_NAME=elastic-1
NODE_HOSTNAME=elastic-1

where:

NODE_IP variable and entire section [elasticsearch-primary] points to IP address of master node.

1.4 Secondary node configuration (optional)

🚧

This step is optional if you don't need HA.

Configuration should be done in analogous way as with master.

[elasticsearch-primary]
x.x.x.x ansible_user=root
...
[elasticsearch:vars]
NODE_IP=x.x.x.x
NODE_DNS_NAME=elastic-1
NODE_HOSTNAME=elastic-1

1.5 Network configuration (optional)

🚧

By default ElasticSearch listens on all network interfaces.

It might be security threat in case cluster servers have public IP addresses.

If you want to change that behaviour you should change property network.host in following configuration file:

AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/elasticsearch/elasticsearch.yml.j2

You can find more details in Network Module documentation.

2. Commit configuration changes

Once you made changes to configuration file you should commit them and push to GIT configuration repository.

$ cd AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services
$ git add elasticsearch/elasticsearch.yml.j2
$ git add inventory
$ git commit -m 'updated configuration of elasticsearch'
$ git push

3. SSH Connectivity check

In order to execute next steps you should make sure that you can connect to the nodes (set in configuration files) via SSH using key-based authentication as root user.

Example

user@localhost:~$ ssh [email protected]
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-47-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
[email protected]:~#

4. Cloning GIT repository with Ansible scripts in $HOME directory

$ cd ~
$ git clone https://repo-ext.sentione.com/public/external-services.git
Cloning into 'external-services'...
Username for 'https://repo-ext.sentione.com': YOUR_GIT_USERNAME
Password for 'https://[email protected]@repo-ext.sentione.com':

5. Master node installation

Installation process is divided in two steps. In the first one we start with installation of Master node.

5.1 Navigate to elasticsearch folder in external-services GIT repository

$ cd ~/external-services/elasticsearch

5.2 Execute master node ansible playbook

$ ansible-playbook \
  -e CONFIG_REPOSITORY_PATH=AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/elasticsearch \
  -i AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/inventory \
  install-elasticsearch-playbook.yaml

Example

$ ansible-playbook \
  -e CONFIG_REPOSITORY_PATH=/home/user/automate-config/production/external-services/elasticsearch \
  -i /home/user/automate-config/production/external-services/inventory \
  install-elasticsearch-playbook.yaml

6. Store ES Certificate

Once you install master node there should be certificate named elastic-stack-ca.pem created in AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/elasticsearch directory.

It's used to secure transmission between applications and ElasticSearch cluster using SSL protocol.

6.1 Commit & Push generated certificate

$ cd AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/
$ git add external-services/elasticsearch/elastic-stack-ca.pem
$ git commit -m'adds elasticsearch certificate for environment ENVIRONMENT_NAME'
$ git push

7. Secondary node installation (optional)

🚧

This step is optional if you don't need HA.

Next step is installation of secondary node and forming two-node cluster.

7.1 Navigate to elasticsearch folder in external-services GIT repository

$ cd ~/external-services/elasticsearch

7.2 Execute secondary node playbook

To do so please issue following commands

$ ansible-playbook \
  -e CONFIG_REPOSITORY_PATH=/home/user/automate-config/production/external-services/elasticsearch \
  -i AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/external-services/inventory \
  install-elasticsearch-secondary-playbook.yaml

8. Test

In order to check if ElasticSearch was successfully installed please issue following curl command.

$ curl -u "es_user_name:es_password" -k https://x.x.x.x:9200/_cluster/health

which should output healthcheck information in JSON format.