AppService status

Monitoring tools installation

Suggest Edits

Document version: 237. Automatically generated.

📘

Loki-stack Loki is an open-source, multi-tenant log aggregation system. It can be used with Grafana and Promtrail to collect and access logs.

The Loki-based logging stack focuses on being light-weight and easy to operate.
Loki-stack helm chart has the following components:

ComponentDescription
lokilog aggregation system designed to store and query logs
prometheussystems monitoring and alerting toolkit
grafanaplatform for visualizing metrics, logs, and traces
alertmanagerservice that handles alerts sent by client applications
promtailagent which ships the contents of logs to a Grafana Loki instance
node-exporterprometheus exporter for hardware and OS metrics
kube-state-metricsservice that listens to the Kubernetes API server and generates metrics

🚧

This external service is optional and not obligatory for successful deployment of Automate platform.

It's possible to install software without persistent volumes, but it means data loss whenever the component storing data will restart. Therefore for bare metal installations, we advise using Local Path Provisioner which will take care of managing local storage in each k8s node. For public cloud Kubernetes installation, there shall be available dedicated storage class options.

Monitoring

Prerequisite

  • Access to the k8s cluster with kubectl and helm.
  • At least 30 GB of available disk space

Conventions

  • CAPITAL_LETTERS_VARIABLE_NAME - fragment which should be replaced with actual values matching particular environment
  • AUTOMATE_CONFIG_REPO_PATH - Path to automate configuration repository (eg. /home/user/automate-config)
  • ENVIRONMENT_NAME - Environment name (eg. production)

Steps

1. Set up helm repo

1.1 Add helm repository

helm repo add sentione-hub https://charts.sentione.com/repository/helm --username USERNAME --password PASSWORD

1.2 Fetch repository data

helm repo update

2. Create a dedicated namespaces

It is recommended to run SentiOne Automate platform in it's dedicated logical Kubernetes space called namespace. To create namespace please use kubectl command.

2.1 For loki-stack

```shell
# creating namespace named 'loki-stack'
$ kubectl create namespace loki-stack
```

```shell
# verifying if namespace has been created
$ kubectl get namespaces
```

2.2 For local path provisioner

As on example below we are installing the software in dedicated local-path-storage namespace then it needs to be created in:

```sh
# creating namespace named 'local-path-storage'
$ kubectl create namespace local-path-storage
```

3. Add a docker registry secret

Kubernetes cluster pulls docker images from harbor-vm-proxy.sentione.com repository. By default, the registry is locked and needs authentication. The object type secret docker-registry is created to previously created Kubernetes namespace to authenticate.

# creating secret type object named 'sec-harbor-vm-proxy.sentione.com' - enabling access to SentiOne docker registry
$ kubectl create secret \
docker-registry sec-harbor-vm-proxy.sentione.com \
--docker-server=harbor-vm-proxy.sentione.com \
--docker-username=USERNAME \
--docker-password=PASSWORD \
[email protected] \
--namespace local-path-storage

3.1 Add a docker registry secret for kubernetes event exporter

kubectl -n local-path-storage create secret docker-registry sec-harbor-vm-proxy.sentione.com --docker-server=https://harbor-vm-proxy.sentione.com --docker-username=docker-RO --docker-password=XXX

4. Local-storage installation

4.1 Run install

helm install local-path-storage --namespace local-path-storage --create-namespace sentione-hub/local-path-provisioner --version 0.0.24-dev

5. Loki-stack installation

5.1 Values for loki-stack installation

Prepare the loki-stack-pv-values.yaml for setting persistent volume configuration. storageClass or storageClassName values can be adjusted when using other than loki storageClass options.
The file can be found in AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/cluster-essentials/monitoring/loki-stack

loki:
  persistence:
    enabled: true
    storageClassName: local-path


grafana:
  persistence:
    enabled: true
    storageClassName: local-path


prometheus:
  alertmanager:
    persistentVolume:
      enabled: true
      storageClass: "local-path"
  
  server:
    persistentVolume:
      enabled: true
      storageClass: "local-path"

The default consumption of PV storage (after installation) could be as follows:

  • loki-stack/loki-grafana 10GB
  • loki-stack/storage-loki 10GB
  • loki-stack/loki-prometheus-server 8GB
  • loki-stack/loki-prometheus-alertmanager 2GB

Please, remember it depends on kubernetes cluster size and retention setting.

5.2 Run the installation

helm -n loki-stack upgrade --install --wait --debug '--timeout=30m' -f loki-stack-pv-values.yaml loki sentione-hub/loki-stack  --version 2.8.5

6 kubernetes event exporter installation

helm -n loki-stack upgrade --install --wait --debug '--timeout=30m' kubernetes-event-exporter sentione-hub/kubernetes-event-exporter  --version 2.1.6

7 Grafana post-configuration

7.1 Expose grafana on public URL (ingress)

Here is a sample ingress configuration for exposing grafana web service:
The file can be found as a 'grafana-ingress-example.yaml' in AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/cluster-essentials/ingress-nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana.domain
  namespace: loki-stack

spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - grafana.domain
  rules:
  - host: grafana.domain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: loki-grafana
            port:
              number: 80

7.2 Extract Grafana's admin login credentials

To fetch password for grafana admin user run the following command:

kubectl get secret --namespace loki-stack loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

7.3 Grafana built-in dashboards for metrics and log browsing

Currently the are several general dashboards provided in the grafana, those could be different for different envs:

For example:

  • loki - logs / app (logs from the application)
  • loki - logs / k8s events (kubernetes events logs)
  • loki - logs / pod (logs from the pods)
  • prometheus - deployment resources usage (load of the resources during
    deployments)
  • prometheus - kubernetes pods (resources usage for pods)
  • Restarts (number of restarts for each node)

Grafana & Loki-stack monitoring can be found here

Updated 5 days ago