Monitoring tools installation
Document version: 244. Automatically generated.
Loki-stack Loki is an open-source, multi-tenant log aggregation system. It can be used with Grafana and Promtrail to collect and access logs.
The Loki-based logging stack focuses on being light-weight and easy to operate.
Loki-stack helm chart has the following components:
Component Description loki log aggregation system designed to store and query logs prometheus systems monitoring and alerting toolkit grafana platform for visualizing metrics, logs, and traces alertmanager service that handles alerts sent by client applications promtail agent which ships the contents of logs to a Grafana Loki instance node-exporter prometheus exporter for hardware and OS metrics kube-state-metrics service that listens to the Kubernetes API server and generates metrics
This external service is optional and not obligatory for successful deployment of Automate platform.
It's possible to install software without persistent volumes, but it means data loss whenever the component storing data will restart. Therefore for bare metal installations, we advise using Local Path Provisioner which will take care of managing local storage in each k8s node. For public cloud Kubernetes installation, there shall be available dedicated storage class options.
Prerequisite
- Access to the k8s cluster with kubectl and helm.
- At least 30 GB of available disk space
Conventions
CAPITAL_LETTERS_VARIABLE_NAME
- fragment which should be replaced with actual values matching particular environmentAUTOMATE_CONFIG_REPO_PATH
- Path to automate configuration repository (eg./home/user/automate-config
)ENVIRONMENT_NAME
- Environment name (eg.production
)
Steps
1. Set up helm repo
1.1 Add helm repository
helm repo add sentione-hub https://charts.sentione.com/repository/helm --username USERNAME --password PASSWORD
1.2 Fetch repository data
helm repo update
2. Create a dedicated namespaces
It is recommended to run SentiOne Automate platform in it's dedicated logical Kubernetes space called namespace. To create namespace please use kubectl
command.
2.1 For loki-stack
```shell
# creating namespace named 'loki-stack'
$ kubectl create namespace loki-stack
```
```shell
# verifying if namespace has been created
$ kubectl get namespaces
```
2.2 For local path provisioner
As on example below we are installing the software in dedicated local-path-storage
namespace then it needs to be created in:
```sh
# creating namespace named 'local-path-storage'
$ kubectl create namespace local-path-storage
```
3. Add a docker registry secret
Kubernetes cluster pulls docker images from harbor-vm-proxy.sentione.com
repository. By default, the registry is locked and needs authentication. The object type secret docker-registry
is created to previously created Kubernetes namespace to authenticate.
# creating secret type object named 'sec-harbor-vm-proxy.sentione.com' - enabling access to SentiOne docker registry
$ kubectl create secret \
docker-registry sec-harbor-vm-proxy.sentione.com \
--docker-server=harbor-vm-proxy.sentione.com \
--docker-username=USERNAME \
--docker-password=PASSWORD \
[email protected] \
--namespace local-path-storage
3.1 Add a docker registry secret for kubernetes event exporter
kubectl -n local-path-storage create secret docker-registry sec-harbor-vm-proxy.sentione.com --docker-server=https://harbor-vm-proxy.sentione.com --docker-username=docker-RO --docker-password=XXX
4. Local-storage installation
4.1 Run install
helm install local-path-storage --namespace local-path-storage --create-namespace sentione-hub/local-path-provisioner --version 0.0.24-dev
5. Loki-stack installation
5.1 Values for loki-stack installation
Prepare the loki-stack-pv-values.yaml
for setting persistent volume configuration. storageClass
or storageClassName
values can be adjusted when using other than loki storageClass options.
The file can be found in AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/cluster-essentials/monitoring/loki-stack
loki:
persistence:
enabled: true
storageClassName: local-path
grafana:
persistence:
enabled: true
storageClassName: local-path
prometheus:
alertmanager:
persistentVolume:
enabled: true
storageClass: "local-path"
server:
persistentVolume:
enabled: true
storageClass: "local-path"
The default consumption of PV storage (after installation) could be as follows:
- loki-stack/loki-grafana 10GB
- loki-stack/storage-loki 10GB
- loki-stack/loki-prometheus-server 8GB
- loki-stack/loki-prometheus-alertmanager 2GB
Please, remember it depends on kubernetes cluster size and retention setting.
5.2 Run the installation
helm -n loki-stack upgrade --install --wait --debug '--timeout=30m' -f loki-stack-pv-values.yaml loki sentione-hub/loki-stack --version 2.8.5
6 kubernetes event exporter installation
helm -n loki-stack upgrade --install --wait --debug '--timeout=30m' kubernetes-event-exporter sentione-hub/kubernetes-event-exporter --version 2.1.6
7 Grafana post-configuration
7.1 Expose grafana on public URL (ingress)
Here is a sample ingress configuration for exposing grafana web service:
The file can be found as a 'grafana-ingress-example.yaml' in AUTOMATE_CONFIG_REPO_PATH/ENVIRONMENT_NAME/cluster-essentials/ingress-nginx
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana.domain
namespace: loki-stack
spec:
ingressClassName: nginx
tls:
- hosts:
- grafana.domain
rules:
- host: grafana.domain
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: loki-grafana
port:
number: 80
7.2 Extract Grafana's admin login credentials
To fetch password for grafana admin user run the following command:
kubectl get secret --namespace loki-stack loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
7.3 Grafana built-in dashboards for metrics and log browsing
Currently the are several general dashboards provided in the grafana, those could be different for different envs:
For example:
- loki - logs / app (logs from the application)
- loki - logs / k8s events (kubernetes events logs)
- loki - logs / pod (logs from the pods)
- prometheus - deployment resources usage (load of the resources during
deployments) - prometheus - kubernetes pods (resources usage for pods)
- Restarts (number of restarts for each node)
Grafana & Loki-stack monitoring can be found here
Updated 12 days ago