Automate

Upgrade tasks

Suggest Edits

Document version: 214. Automatically generated.

Document lists important changes to Automate platform which affect existing deployments.

Each change is provided with migration guide which has to be applied in order to keep platform running correctly.

Releases

Ordered from latest to oldest

214

1. Reddit Domain Category

Issue ID: DEV-14830

WHERE?
Environments where Listen crawlers are used

Apply SQL script on chimeo (new-web) database AFTER THE DEPLOY.

do $$
    declare
        domain_exists boolean := false;
    begin

        select true from urldomain where host_name = 'www.reddit.com' INTO domain_exists;

        if not found then
            INSERT INTO urldomain (id, ip, allow_redirect_to_host, allow_subdomains, archived, automatic_learn_count, charset, commentsapiname, crawl, created_at, custom_cookie, deactivation_date, from_auto_search, from_blog_search, group_id, host_name, important, kindness, lang, learn_by_generic_to_configurable, learning_date, learning_flow_state, manual_profile, max_depth, modified_at, name, never_crawl, no_statements_sites_to_learn, no_statements_sites_to_reset, notes, path_after_host, posts_page_link, profile, robotstxt, seo_moz_domain_authority, topics_page_link, user_agent, ignore_encoding, average_posts_per_month, monthly_uniqueurls, average_page_views, average_time_on_site, bounce_rate, data_month, seo_moz_page_authority, estimated_visits, sites_with_statements, sites_without_statements, fraction_of_whole_host, always_try_commentsapi, banned_hosts, ignore_robotstxt, version, manual_kindness, excluded_subdomains, excluded_paths_after_host, last_modified_by, accepted_languages, too_long_statements_error, max_posts_per_page, modified_by_user_at, proxy_host, proxy_port, problem, statistics_updated_at, queue_restart_interval, allow_different_top_level_domain_links, search_profile, only_private, country, download_mode, inactivity_reason) VALUES (74, null, null, false, false, null, 'UTF-8', null, false, '2023-03-15 12:45:33.403000 +00:00', null, null, null, null, 44, 'www.reddit.com', 0, 10000, 'en', false, null, null, false, null, '2023-03-15 13:25:45.791000 +00:00', 'https://www.reddit.com', false, null, null, null, null, null, 'eNp9k9tugzAMht+F6z1B76quSGithlh3kKZdWMFQq8FBiWk7TXv3BbK1AardIOX7fbb5SiwqY8u3HGSfLJLkLqHRgwUtg85uwwJbDQq91JOgliAYGUMne2NnYBRQGR+RJTayQkrj/SRWoMtZyMBXozDvHxdhR6JvxHkhPLkIt8aR0DG2ZOzEgh6RGiZGzielxn+e/Chj4YCfJz/ckMOFksK8H24oVLN3z6HGmGriQ4E1GZ7SlelajeHdeq+NZ1HvvYnbWWBXGdv4mg3/OQPXLlRYEZfbTgv5SBk/MhZDecmiAu2wP4bcYkXnYN1gSRC3sb0COIKADXaytwhlStZJbpxMthjUDfwj9oKb0em+Ai2mBxzwzrSkopEM9QN3oPuM7tKhvz0FEk7qAkEpbGUZzsS9khc7mfh1TkzTs3QY7+9s/axSa5o1K1MS13FF2bzIrHy2evT39FtLSXuwPivduevWrF8h16NuzsWUaaOGTfev7x904FAw', null, null, null, null, false, null, null, null, null, null, null, null, null, null, null, null, false, null, false, 0, false, null, null, '[email protected]', null, false, null, '2023-03-15 13:25:45.789000 +00:00', null, null, false, null, null, false, null, false, null, 'HttpClientMode', null);
        else
            UPDATE urldomain SET group_id = 44 WHERE host_name = 'www.reddit.com';
        end if;
    end $$

2. hookServer.publicUrl config parameter change

📘

This change doesn't affect existing environments, so nothing has to be changed.

📘

This change has been also backported to Automate 213.0.31.

Existing config property used for defining public URL of Sentione's hook-server has become polymorphic.
It can now both accept single string value as well as list of string values. Thanks to this change, hook-server can be exposed on multiple different URLs.

📘

Some sources (like Twitter) use this property to register webhooks subscription. In this scenario - first URL from the list is going to be used.

To make a use of this property modify following key in sentione.config.configYaml in [ENV_NAME]/kubernetes/helm/automate/values.yaml using one of two possible options:

  1. as string
hookServer:
    publicUrl: "https://sentione-webhooks-url.com"
  1. as list of strings
hookServer:
    publicUrl: ["https://sentione-webhooks-url.com", "https://alternative-sentione-webhooks-url.com"]

213

1. Unsync NLU after the deploy

Issue: DEV-14857

Due to the RedisAI deployment, all NLUs will have to be retrained. To mark all NLUs "Not ready", following SQL script on DB chatbots-admin
has to be triggered. Please run it only when deployment is verified as successful, because it cannot be rolled back.

UPDATE nlu_models SET status = 'OutSync';

2. Upgrade ElasticSearch cluster from 6.8 to 7.17.9

This is major upgrade of document database used for storing chatbot transcriptions.

More details about the upgrade steps soon.

3. Install RedisAI

There are some major changes to NLU services which now require installation of RedisAI cluster.

More details about the upgrade steps soon.

212

1. Conditional update script bug has been fixed

Issue ID: DEV-14883

Mentions from social media like Instagram & Facebook were affected by a bug which could cause certain properties to be wiped out during metadata refresh.

In order for fix to be effective - following manual steps has to be followed.

Steps

1. Cloning GIT repository with ElasticSearch Ansible scripts in $HOME directory
$ cd ~
$ git clone https://repo-ext.sentione.com/public/external-services.git
Cloning into 'external-services'...
Username for 'https://repo-ext.sentione.com': YOUR_GIT_USERNAME
Password for 'https://[email protected]@repo-ext.sentione.com':

🚧

If you already cloned the repo - please pull latest changes!

2. Navigate to $HOME/external-services/elasticsearch/6.8.23/scripts
$ cd ~/external-services/elasticsearch/6.8.23/scripts
3. Run storeScripts.sh
$ ./storeScripts.sh ELASTICSEARCH_HOSTNAME ELASTICSEARCH_PORT ELASTICSEARCH_USERNAME ELASTICSEARCH_PASSWORD ELASTICSEARCH_CERTIFICATE_PATH

Example

$ ./storeScripts.sh elastic-master-host1 9200 esuser espassword ~/automate-config/production/external-services/elasticsearch/elastic-stack-ca.pem
Storing commonValuesUpdateScriptPain.painless...
Storing conditionalUpdate.painless...
Storing dayOfWeekIntScriptFieldPain.painless...
Storing dayOfWeekScriptFieldPain.painless...
Storing hourOfDayFilterPain.painless...
Storing hourOfDayScriptFieldPain.painless...
Storing opinionUpdate.painless...
Storing removeAndAddTagsScriptPain.painless...

211

1. New application channels-connector

Channels connector is new optional application.

Steps

Please follow steps below if you want to turn it on.

1. Enable application deployment

By default, the channels-connector application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml

sentione:
  channelsConnector: 
    enable: true
2. Expose Channels connector bot on public URL

Channels connector needs to be accessible from the outside.

This means that this application needs to be accessible from public internet in order to work properly.

2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: envname-automate-ingress
spec:
  tls: # add spec.tls section
  - hosts: # include URIs to be secured by the certificate
    - channels-connector-envname.automate.example.com
    secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
  rules:
  - host: channels-connector-envname.automate.example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: channels-connector
              port:
                number: 5782
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml

210

1. New application ms-teams-bot

MS Teams bot is new optional application.

Steps

Please follow steps below if you want to turn it on.

1. Enable application deployment

By default, the ms-teams-bot application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml

sentione:
  msTeamsBot: 
    enable: true
2. Expose MS Teams bot on public URL

MS Teams bot application receives and sends webhooks from/to Microsoft Teams.

This means that this application needs to be accessible from public internet in order to work properly.

2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: envname-automate-ingress
spec:
  tls: # add spec.tls section
  - hosts: # include URIs to be secured by the certificate
    - ms-teams-bot-envname.automate.example.com
    secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
  rules:
  - host: ms-teams-bot-envname.automate.example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: ms-teams-bot
              port:
                number: 5770
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml

208

1. New application hooks-server

React Hooks Server is new optional application used by WhatsApp Bots.

📘

hooks-server application has been also backported to 207.0.17.

Steps

Please follow steps below if you want to turn it on

1. Enable application deployment

By default, the hooks-server application is disabled and won't be deployed. To enable it modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml

sentione:
  hooksServer: 
    enable: true
2. Expose hook server on public URL

Hook server application receives webhooks coming from Third party services like Twilio, Facebook, Twitter etc.

This means that this application needs to be accessible from public internet in order to work properly.

2.1 Add following entry to [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: envname-automate-ingress
spec:
  tls: # add spec.tls section
  - hosts: # include URIs to be secured by the certificate
    - hooks-server-envname.automate.example.com
    secretName: tls-wildcard-example-com-secret # TLS certificate secret name that will be added in the next step
  rules:
  - host: hook-server-envname.automate.example.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: hooks-server
              port:
                number: 8069
2.2 Install ingress configuration on Kubernetes cluster & configure any proxy servers in front of it.
kubectl apply -f [ENV_NAME]/kubernetes/ingress/automate_ingress.yaml
3. Add hookServer configuration section to config.yaml

Modify sentione.config.configYaml key in [ENV_NAME]/kubernetes/helm/automate/values.yaml by adding following section:

onpremise:
  hookServer:
     publicUrl: https://PUT_HOOKS_SERVER_PUBLIC_URL
     automate:
         # Please make sure it matches Chatbots config key: chatbots.thread-coordinator.hooks.otp-secret
         otpSecret: PUT_OTP_SECRET_HERE

2. New database chatbots-thread-coordinator

System now requires additional database for the purpose of thread-coordinator application.

Steps

1. Create new database in PostgreSQL Server by issuing following SQL command
CREATE DATABASE "chatbots-thread-coordinator";

3. New application thread-coordinator

Thread coordinator is new optional application used by WhatsApp Bots.

Steps

Please follow steps below if you want to turn it on

1. Enable application deployment

Modify following configuration keys in [ENV_NAME]/kubernetes/helm/automate/values.yaml

chatbots:
  apps:
    threadCoordinator: true
2. Add thread coordinator configuration section

Add following section to [ENV_NAME]/kubernetes/helm/automate/chatbots/common.conf

chatbots.thread-coordinator {
  db {
    url: "jdbc:postgresql://database-host:5432/chatbots-thread-coordinator",
    user: "PUT_USERNAME_HERE",
    password: "PUT_PASSWORD_HERE"
    max-connections: 20
  }
  
  hooks {
    hook-receivers-urls: ["https://REACT_HOOKS_SERVER_PUBLIC_URL/automate"]
    hooks-request-timeout: 30s
    # Please make sure it matches with OTP Secret set in SentiOne's config.yaml key: hookServer.automate.otpSecret
    otp-secret: "PUT_OTP_SECRET_HERE"
  }
}

206

1. Elasticsearch script

We have added two new fields into to Statement schema, in order to update your current ES cluster there is a need to run a script from our repository.

Steps

1. Cloning GIT repository with Ansible scripts
git clone https://repo-ext.sentione.com/public/external-services.git
Cloning into 'external-services'...
Username for 'https://repo-ext.sentione.com': YOUR_GIT_USERNAME
Password for 'https://[email protected]@repo-ext.sentione.com': YOUR_GIT_PASSWORD
2. Run existing mappings script

After pulling changes in elasticsearch repo run script with parameters

external-services/elasticsearch/6.8.23/indexScripts/updateExistingMappingInPlace/9-add-author-category.sh ELASTIC_HOST PORT [USERNAME] [PASSWORD]

3. Run index template script [NEW]

Step added: 30.11.2022

Run script with parameters

external-services/elasticsearch/6.8.23/indexScripts/updateAllIndexTemplates.sh ELASTIC_HOST PORT [USERNAME] [PASSWORD]

205

1. Docker image registry URL change

Docker image registry has changed from dockerhub.sentione.com to harbor-vm-proxy.sentione.com

In order to upgrade to new version of the platform you'll have to create new secret for pulling images from the new registry.

Default secret name for pulling images is set inside automate's HELM package values.yaml and have following values:

nlu:
    imagePullSecrets:
    - name: "sec-harbor-vm-proxy.sentione.com"
chatbots:
    imagePullSecrets:
    - name: "sec-harbor-vm-proxy.sentione.com"
sentione:
    imagePullSecrets:
    - name: "sec-harbor-vm-proxy.sentione.com"

If you override these values in your deployment you'll have to update commands below accordingly.

Steps

  1. Please execute following commands on your Kubernetes clusters:
# creating secret type object named 'sec-harbor-vm-proxy.sentione.com' - enabling access to SentiOne docker registry
kubectl create secret \
docker-registry sec-harbor-vm-proxy.sentione.com \
--docker-server=harbor-vm-proxy.sentione.com \
--docker-username=USERNAME \
--docker-password=PASSWORD \
[email protected] \
--namespace sentione
  1. Verify secret has been successfully created
# veryfying if secret type object has been created in sentione namespace
kubectl get secrets -n sentione sec-harbor-vm-proxy.sentione.com

NAME                               TYPE                             DATA   AGE
sec-harbor-vm-proxy.sentione.com   kubernetes.io/dockerconfigjson   1      2d17h

2. Docker registry key structure has changed.

Structure & Name of following keys from values.yaml has changed:

  • chatbots.docker_registry
  • sentione.docker_registry
  • nlu.docker_registry

Steps

  1. If configuration for your environment overwrites default docker registry (eg. because of using self-hosted proxy),
    please update your values.yaml accordingly to changes listed below.

Chatbots

  • Before:
    chatbots.docker_registry: custom-charts-repo.com
  • After:
    chatbots.dockerRegistry:
  host: custom-charts-repo.com
  path:
    application: /chatbots

SentiOne

  • Before:
    sentione.docker_registry: custom-charts-repo.com
  • After:
    sentione.dockerRegistry:
  host: custom-charts-repo.com
  path:
    application: /sentione
    thirdParty: /sentione

NLU

  • Before:
    nlu.docker_registry: custom-charts-repo.com
  • After:
    nlu.dockerRegistry:
  host: custom-charts-repo.com
  path:
    application: /nlu

3. Webchat is now using Redis

Redis is used for delivering websocket messages across different instances of webchat application. Webchat connects to Redis using host and port defined in chatbots.concurrency secion of config. It's the same config that is already used by gateway and dialog manager applications.

Default config:

chatbots.concurrency {
  redis {
    host: "redis"
    port: 6379
  }
}

You can override default config by adding this section to chatbots.config.commonConf key in customValues.yaml

4. Webchat is now using postgres database

Database has to be created manually by executing script:

CREATE DATABASE "chatbots-web-chat";

Default config:

chatbots.web-chat.db {
  url: "jdbc:postgresql://db:5432/chatbots-web-chat",
  user: "postgres",
  password: "root"
  max-connections: 20
}

You can override default config by adding this section to chatbots.config.commonConf key in customValues.yaml

202

  • Text to speech integration (optional - does not affect on-premise deployments)
    • There's new experimental feature of integrating with Text To Speech service
    • To set it up please add following section to chatbots.config.commonConf key in customValues.yaml
      chatbots.admin.text-to-speech {
  base-url: "http://text-to-speech-service-host:PORT"
    services {
      pl: "techmo-dev" # Endpoint name for polish language
      en: "techmo-dev-en" # Endpoint name for english language
    }
 }
  • Speech to text integration (optional - does not affect on-premise deployments)
    • There's new experimental feature of integrating with Speech To Text service (via GRPC)
    • To set it up please add following section to chatbots.config.commonConf key in customValues.yaml
    chatbots.admin.speech-to-text {
  services {
    pl { # Configuration for ASR service for polish language
        hostname: "HOSTNAME" # Hostname of GRPC service hosting Speech To Text endpoint 
        port: 6881 # Port of GRPC service hosting Speech to Text Endpoint
    }
    
    en { # Configuration for ASR service for english language
         hostname: "HOSTNAME"
         port: 6882
    }
    
    es { # Configuration for ASR service for spanish language
         hostname: "HOSTNAME"
         port: 6883
    }
  }
}

199

  • SentiOne application (web, refinery etc.) is added as optional dependency to automate HELM package
    • On environments where SentiOne React was not needed you'll have to add following section to values.yaml
      sentione:
  enabled: false
    • This change was also backported to previous releases:
      • 197.0.9
      • 198.0.3
  • Installation method changes. Instead of installing individual components (nlu, sentione, chatbots) you should install one aggregated package called Automate. More details in INSTALLATION.md

195

Updated 5 days ago