Upgrade tasks
Document version: 214. Automatically generated.
Document lists important changes to Automate platform which affect existing deployments.
Each change is provided with migration guide which has to be applied in order to keep platform running correctly.
Releases
Ordered from latest to oldest
214
1. Reddit Domain Category
Issue ID: DEV-14830
WHERE?
Environments where Listen crawlers are used
Apply SQL script on chimeo
(new-web) database AFTER THE DEPLOY.
do $$
declare
domain_exists boolean := false;
begin
select true from urldomain where host_name = 'www.reddit.com' INTO domain_exists;
if not found then
INSERT INTO urldomain (id, ip, allow_redirect_to_host, allow_subdomains, archived, automatic_learn_count, charset, commentsapiname, crawl, created_at, custom_cookie, deactivation_date, from_auto_search, from_blog_search, group_id, host_name, important, kindness, lang, learn_by_generic_to_configurable, learning_date, learning_flow_state, manual_profile, max_depth, modified_at, name, never_crawl, no_statements_sites_to_learn, no_statements_sites_to_reset, notes, path_after_host, posts_page_link, profile, robotstxt, seo_moz_domain_authority, topics_page_link, user_agent, ignore_encoding, average_posts_per_month, monthly_uniqueurls, average_page_views, average_time_on_site, bounce_rate, data_month, seo_moz_page_authority, estimated_visits, sites_with_statements, sites_without_statements, fraction_of_whole_host, always_try_commentsapi, banned_hosts, ignore_robotstxt, version, manual_kindness, excluded_subdomains, excluded_paths_after_host, last_modified_by, accepted_languages, too_long_statements_error, max_posts_per_page, modified_by_user_at, proxy_host, proxy_port, problem, statistics_updated_at, queue_restart_interval, allow_different_top_level_domain_links, search_profile, only_private, country, download_mode, inactivity_reason) VALUES (74, null, null, false, false, null, 'UTF-8', null, false, '2023-03-15 12:45:33.403000 +00:00', null, null, null, null, 44, 'www.reddit.com', 0, 10000, 'en', false, null, null, false, null, '2023-03-15 13:25:45.791000 +00:00', 'https://www.reddit.com', false, null, null, null, null, null, 'eNp9k9tugzAMht+F6z1B76quSGithlh3kKZdWMFQq8FBiWk7TXv3BbK1AardIOX7fbb5SiwqY8u3HGSfLJLkLqHRgwUtg85uwwJbDQq91JOgliAYGUMne2NnYBRQGR+RJTayQkrj/SRWoMtZyMBXozDvHxdhR6JvxHkhPLkIt8aR0DG2ZOzEgh6RGiZGzielxn+e/Chj4YCfJz/ckMOFksK8H24oVLN3z6HGmGriQ4E1GZ7SlelajeHdeq+NZ1HvvYnbWWBXGdv4mg3/OQPXLlRYEZfbTgv5SBk/MhZDecmiAu2wP4bcYkXnYN1gSRC3sb0COIKADXaytwhlStZJbpxMthjUDfwj9oKb0em+Ai2mBxzwzrSkopEM9QN3oPuM7tKhvz0FEk7qAkEpbGUZzsS9khc7mfh1TkzTs3QY7+9s/axSa5o1K1MS13FF2bzIrHy2evT39FtLSXuwPivduevWrF8h16NuzsWUaaOGTfev7x904FAw', null, null, null, null, false, null, null, null, null, null, null, null, null, null, null, null, false, null, false, 0, false, null, null, '[email protected]', null, false, null, '2023-03-15 13:25:45.789000 +00:00', null, null, false, null, null, false, null, false, null, 'HttpClientMode', null);
else
UPDATE urldomain SET group_id = 44 WHERE host_name = 'www.reddit.com';
end if;
end $$
2. hookServer.publicUrl config parameter change
This change doesn't affect existing environments, so nothing has to be changed.
This change has been also backported to Automate 213.0.31.
Existing config property used for defining public URL of Sentione's hook-server has become polymorphic.
It can now both accept single string value as well as list of string values. Thanks to this change, hook-server can be exposed on multiple different URLs.
Some sources (like Twitter) use this property to register webhooks subscription. In this scenario - first URL from the list is going to be used.
To make a use of this property modify following key in sentione.config.configYaml
in [ENV_NAME]/kubernetes/helm/automate/values.yaml
using one of two possible options:
- as string
hookServer:
publicUrl: "https://sentione-webhooks-url.com"
- as list of strings
hookServer:
publicUrl: ["https://sentione-webhooks-url.com", "https://alternative-sentione-webhooks-url.com"]
213
1. Unsync NLU after the deploy
Issue: DEV-14857
Due to the RedisAI deployment, all NLUs will have to be retrained. To mark all NLUs "Not ready", following SQL script on DB chatbots-admin
has to be triggered. Please run it only when deployment is verified as successful, because it cannot be rolled back.
UPDATE nlu_models SET status = 'OutSync';
2. Upgrade ElasticSearch cluster from 6.8 to 7.17.9
This is major upgrade of document database used for storing chatbot transcriptions.
More details about the upgrade steps soon.
3. Install RedisAI
There are some major changes to NLU services which now require installation of RedisAI cluster.
More details about the upgrade steps soon.
212
1. Conditional update script bug has been fixed
Issue ID: DEV-14883
Mentions from social media like Instagram & Facebook were affected by a bug which could cause certain properties to be wiped out during metadata refresh.
In order for fix to be effective - following manual steps has to be followed.
Steps
1. Cloning GIT repository with ElasticSearch Ansible scripts in $HOME
directory
$HOME
directory$ cd ~
$ git clone https://repo-ext.sentione.com/public/external-services.git
Cloning into 'external-services'...
Username for 'https://repo-ext.sentione.com': YOUR_GIT_USERNAME
Password for 'https://[email protected]@repo-ext.sentione.com':
If you already cloned the repo - please pull latest changes!
2. Navigate to $HOME/external-services/elasticsearch/6.8.23/scripts
$ cd ~/external-services/elasticsearch/6.8.23/scripts
3. Run storeScripts.sh
$ ./storeScripts.sh ELASTICSEARCH_HOSTNAME ELASTICSEARCH_PORT ELASTICSEARCH_USERNAME ELASTICSEARCH_PASSWORD ELASTICSEARCH_CERTIFICATE_PATH
Example
$ ./storeScripts.sh elastic-master-host1 9200 esuser espassword ~/automate-config/production/external-services/elasticsearch/elastic-stack-ca.pem
Storing commonValuesUpdateScriptPain.painless...
Storing conditionalUpdate.painless...
Storing dayOfWeekIntScriptFieldPain.painless...
Storing dayOfWeekScriptFieldPain.painless...
Storing hourOfDayFilterPain.painless...
Storing hourOfDayScriptFieldPain.painless...
Storing opinionUpdate.painless...
Storing removeAndAddTagsScriptPain.painless...
211
1. New application channels-connector
channels-connector
Channels connector is new optional application.
Steps
Please follow steps below if you want to turn it on.
1. Enable application deployment
By default, the channels-connector
application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml
sentione:
channelsConnector:
enable: true
2. Expose Channels connector bot on public URL
Channels connector needs to be accessible from the outside.
This means that this application needs to be accessible from public internet in order to work properly.
2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
[ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: envname-automate-ingress
spec:
tls: # add spec.tls section
- hosts: # include URIs to be secured by the certificate
- channels-connector-envname.automate.example.com
secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
rules:
- host: channels-connector-envname.automate.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: channels-connector
port:
number: 5782
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
210
1. New application ms-teams-bot
ms-teams-bot
MS Teams bot is new optional application.
Steps
Please follow steps below if you want to turn it on.
1. Enable application deployment
By default, the ms-teams-bot
application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml
sentione:
msTeamsBot:
enable: true
2. Expose MS Teams bot on public URL
MS Teams bot application receives and sends webhooks from/to Microsoft Teams.
This means that this application needs to be accessible from public internet in order to work properly.
2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
[ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: envname-automate-ingress
spec:
tls: # add spec.tls section
- hosts: # include URIs to be secured by the certificate
- ms-teams-bot-envname.automate.example.com
secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
rules:
- host: ms-teams-bot-envname.automate.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ms-teams-bot
port:
number: 5770
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
208
1. New application hooks-server
hooks-server
React Hooks Server is new optional application used by WhatsApp Bots.
hooks-server application has been also backported to 207.0.17.
Steps
Please follow steps below if you want to turn it on
1. Enable application deployment
By default, the hooks-server
application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml
sentione:
hooksServer:
enable: true
2. Expose hook server on public URL
Hook server application receives webhooks coming from Third party services like Twilio, Facebook, Twitter etc.
This means that this application needs to be accessible from public internet in order to work properly.
2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
[ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: envname-automate-ingress
spec:
tls: # add spec.tls section
- hosts: # include URIs to be secured by the certificate
- hooks-server-envname.automate.example.com
secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
rules:
- host: hook-server-envname.automate.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hooks-server
port:
number: 8069
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
3. Add hookServer configuration section to config.yaml
Modify sentione.config.configYaml
key in [ENV_NAME]/kubernetes/helm/automate/values.yaml
by adding following section:
onpremise:
hookServer:
publicUrl: https://PUT_HOOKS_SERVER_PUBLIC_URL
automate:
# Please make sure it matches Chatbots config key: chatbots.thread-coordinator.hooks.otp-secret
otpSecret: PUT_OTP_SECRET_HERE
2. New database chatbots-thread-coordinator
chatbots-thread-coordinator
System now requires additional database for the purpose of thread-coordinator
application.
Steps
1. Create new database in PostgreSQL Server by issuing following SQL command
CREATE DATABASE "chatbots-thread-coordinator";
3. New application thread-coordinator
thread-coordinator
Thread coordinator is new optional application used by WhatsApp Bots.
Steps
Please follow steps below if you want to turn it on
1. Enable application deployment
Modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml
chatbots:
apps:
threadCoordinator: true
2. Add thread coordinator configuration section
Add following section to [ENV_NAME]/kubernetes/helm/automate/chatbots/common.conf
chatbots.thread-coordinator {
db {
url: "jdbc:postgresql://database-host:5432/chatbots-thread-coordinator",
user: "PUT_USERNAME_HERE",
password: "PUT_PASSWORD_HERE"
max-connections: 20
}
hooks {
hook-receivers-urls: ["https://REACT_HOOKS_SERVER_PUBLIC_URL/automate"]
hooks-request-timeout: 30s
# Please make sure it matches with OTP Secret set in SentiOne's config.yaml key: hookServer.automate.otpSecret
otp-secret: "PUT_OTP_SECRET_HERE"
}
}
206
1. Elasticsearch script
We have added two new fields into to Statement
schema, in order to update your current ES cluster there is a need to run a script from our repository.
Steps
1. Cloning GIT repository with Ansible scripts
git clone https://repo-ext.sentione.com/public/external-services.git
Cloning into 'external-services'...
Username for 'https://repo-ext.sentione.com': YOUR_GIT_USERNAME
Password for 'https://[email protected]@repo-ext.sentione.com': YOUR_GIT_PASSWORD
2. Run existing mappings script
After pulling changes in elasticsearch repo run script with parameters
external-services/elasticsearch/6.8.23/indexScripts/updateExistingMappingInPlace/9-add-author-category.sh ELASTIC_HOST PORT [USERNAME] [PASSWORD]
3. Run index template script [NEW]
Step added: 30.11.2022
Run script with parameters
external-services/elasticsearch/6.8.23/indexScripts/updateAllIndexTemplates.sh ELASTIC_HOST PORT [USERNAME] [PASSWORD]
205
1. Docker image registry URL change
Docker image registry has changed from dockerhub.sentione.com to harbor-vm-proxy.sentione.com
In order to upgrade to new version of the platform you'll have to create new secret for pulling images from the new registry.
Default secret name for pulling images is set inside automate's HELM package values.yaml and have following values:
nlu:
imagePullSecrets:
- name: "sec-harbor-vm-proxy.sentione.com"
chatbots:
imagePullSecrets:
- name: "sec-harbor-vm-proxy.sentione.com"
sentione:
imagePullSecrets:
- name: "sec-harbor-vm-proxy.sentione.com"
If you override these values in your deployment you'll have to update commands below accordingly.
Steps
- Please execute following commands on your Kubernetes clusters:
# creating secret type object named 'sec-harbor-vm-proxy.sentione.com' - enabling access to SentiOne docker registry
kubectl create secret \
docker-registry sec-harbor-vm-proxy.sentione.com \
--docker-server=harbor-vm-proxy.sentione.com \
--docker-username=USERNAME \
--docker-password=PASSWORD \
[email protected] \
--namespace sentione
- Verify secret has been successfully created
# veryfying if secret type object has been created in sentione namespace
kubectl get secrets -n sentione sec-harbor-vm-proxy.sentione.com
NAME TYPE DATA AGE
sec-harbor-vm-proxy.sentione.com kubernetes.io/dockerconfigjson 1 2d17h
2. Docker registry key structure has changed.
Structure & Name of following keys from values.yaml has changed:
- chatbots.docker_registry
- sentione.docker_registry
- nlu.docker_registry
Steps
- If configuration for your environment overwrites default docker registry (eg. because of using self-hosted proxy),
please update your values.yaml accordingly to changes listed below.
Chatbots
- Before:
chatbots.docker_registry: custom-charts-repo.com
- After:
chatbots.dockerRegistry: host: custom-charts-repo.com path: application: /chatbots
SentiOne
- Before:
sentione.docker_registry: custom-charts-repo.com
- After:
sentione.dockerRegistry: host: custom-charts-repo.com path: application: /sentione thirdParty: /sentione
NLU
- Before:
nlu.docker_registry: custom-charts-repo.com
- After:
nlu.dockerRegistry: host: custom-charts-repo.com path: application: /nlu
3. Webchat is now using Redis
Redis is used for delivering websocket messages across different instances of webchat application. Webchat connects to Redis using host and port defined in chatbots.concurrency secion of config. It's the same config that is already used by gateway and dialog manager applications.
Default config:
chatbots.concurrency {
redis {
host: "redis"
port: 6379
}
}
You can override default config by adding this section to chatbots.config.commonConf key in customValues.yaml
4. Webchat is now using postgres database
Database has to be created manually by executing script:
CREATE DATABASE "chatbots-web-chat";
Default config:
chatbots.web-chat.db {
url: "jdbc:postgresql://db:5432/chatbots-web-chat",
user: "postgres",
password: "root"
max-connections: 20
}
You can override default config by adding this section to chatbots.config.commonConf key in customValues.yaml
202
- Text to speech integration (optional - does not affect on-premise deployments)
- There's new experimental feature of integrating with Text To Speech service
- To set it up please add following section to chatbots.config.commonConf key in customValues.yaml
chatbots.admin.text-to-speech { base-url: "http://text-to-speech-service-host:PORT" services { pl: "techmo-dev" # Endpoint name for polish language en: "techmo-dev-en" # Endpoint name for english language } }
- Speech to text integration (optional - does not affect on-premise deployments)
- There's new experimental feature of integrating with Speech To Text service (via GRPC)
- To set it up please add following section to chatbots.config.commonConf key in customValues.yaml
chatbots.admin.speech-to-text { services { pl { # Configuration for ASR service for polish language hostname: "HOSTNAME" # Hostname of GRPC service hosting Speech To Text endpoint port: 6881 # Port of GRPC service hosting Speech to Text Endpoint } en { # Configuration for ASR service for english language hostname: "HOSTNAME" port: 6882 } es { # Configuration for ASR service for spanish language hostname: "HOSTNAME" port: 6883 } } }
199
- SentiOne application (web, refinery etc.) is added as optional dependency to automate HELM package
- On environments where SentiOne React was not needed you'll have to add following section to values.yaml
sentione: enabled: false
- This change was also backported to previous releases:
- 197.0.9
- 198.0.3
- On environments where SentiOne React was not needed you'll have to add following section to values.yaml
- Installation method changes. Instead of installing individual components (nlu, sentione, chatbots) you should install one aggregated package called Automate. More details in INSTALLATION.md
195
- Implemented HELM package versioning
- Implemented Installation Guide versioning
- Latest installation instruction can now be found on https://automate.help.sentione.com/docs/installation-manual
Updated 5 days ago