Install Grafana Loki on a local Kubernetes Cluster

by

This guide describes how to install a Grafana Loki instance with Event Exporter and Promtail on a local Kubernetes Cluster (using KinD).

First you will install Docker on Windows, then deploy a Kubernetes Cluster on Docker using Kind. After that you will install Grafana Loki and Event Exporter and Promtail. The last step is executing a Loki query and admire your results!

The solutions used in this guide are:

SolutionFunctionality
DockerA platform for developing, shipping, and running applications inside lightweight, isolated containers.
KubernetesAn open-source system for automating the deployment, scaling, and management of containerized applications.
KubectlA command-line tool to interact with and manage Kubernetes clusters.
KindA tool to run local Kubernetes clusters using Docker containers for easy testing and development.
HelmA package manager for Kubernetes, enabling easier application deployment and management using reusable charts.
PrometheusAn open-source monitoring and alerting toolkit designed for reliability and scalability in cloud environments.
Grafana LokiA log aggregation system designed to work with Grafana for querying, visualizing, and alerting on logs.
Kubernetes Event ExporterA tool for exporting Kubernetes events to various logging and monitoring backends.
PromtailAn agent that collects logs and forwards them to Loki for centralized log management and querying.

Prerequisites

Install Docker for Windows

https://docs.docker.com/desktop/install/windows-install

Get kubectl

Download kubectl (for Windows) and copy to: c:\k8s

Install Kind

Download Kind (for Windows) and move to c:\k8s\kind\kind.exe
Don’t forget to add the file extension .exe

Quick guide available here: https://kind.sigs.k8s.io/docs/user/quick-start/

Add path to system environment variable

Add c:\k8s\kind to environment variable Path.

Create Kubernetes Cluster

Configures and starts a K8s cluster using KinD on the local machine.

Create kind.yaml

c:\k8s\kind.yaml

# This config file contains all config fields with comments
kind: Cluster  
apiVersion: kind.x-k8s.io/v1alpha4  
# 1 control plane node and 3 workers  
nodes:  
# the control plane node config  
- role: control-plane  
# the three workers  
- role: worker  
- role: worker  
- role: worker

cd c:\k8s
kind create cluster --config kind.yaml --name k8s-lab

Install Helm

Role: used to install helm charts (k8s templates)

https://helm.sh/docs/intro/install

winget install Helm.Helm

Could be you have restart cmd.exe to make Helm available/working

Deploy Prometheus

Add repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install

kubectl create namespace prometheus
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack -n prometheus

Get status

kubectl get services --namespace prometheus
kubectl --namespace prometheus get pods -l "release=prometheus"

Port forward

Forward port 3000 from local host to Grafana container by starting a new Command Prompt and execute the following command. Leave this windows open, the port forwarding will be cancelled when you close the window.

kubectl port-forward deployment/prometheus-grafana -n prometheus 3000


Open browser and browse to: http://localhost:3000/login

Default credentials:
User: admin
pass: prom-operator

Default User/Password is defined in chart

Deploy Grafana Loki

Add repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

loki.yaml

c:\k8s\loki.yaml

loki:
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
    max_concurrent: 4
  auth_enabled: false
#gateway:
#  ingress:
#    enabled: true
#    hosts:
#      - host: FIXME
#        paths:
#          - path: /
#            pathType: Prefix

deploymentMode: SimpleScalable

backend:
  replicas: 3
read:
  replicas: 3
write:
  replicas: 3

# Enable minio for storage
minio:
  enabled: true

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

Install

kubectl create namespace loki
helm install --values loki.yaml loki --namespace=loki grafana/loki

Get status

kubectl get services --namespace loki
kubectl --namespace loki get pods -l "release=loki"

Configure Loki Datasource

URL: http://loki-gateway.loki.svc.cluster.local/
This is the name of the container loki-gateway.

Deploy Kubernetes Event Exporter

Role: Forwards all Kubernetes Events to Loki

Add repository

helm repo add bitnami https://charts.bitnami.com/bitnami

eventexporter.yaml

c:\k8s\eventexporter.yaml

config:
  leaderElection: {}
  logLevel: debug
  logFormat: pretty
  metricsNamePrefix: event_exporter_
  receivers:
    - name: "dump"
      file:
        path: "/dev/stdout"
        layout: {}
    - name: "loki"
      loki:
        url: "http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push"
        streamLabels:
          source: kubernetes-event-exporter
          container: kubernetes-event-exporter
  route:
    routes:
      - match:
          - receiver: "dump"
          - receiver: "loki"

Install

kubectl create namespace event-exporter
helm install event-exporter bitnami/kubernetes-event-exporter --values eventexporter.yaml --namespace=event-exporter

Get Status

kubectl get all -l app.kubernetes.io/name=kubernetes-event-exporter --namespace event-exporter

Deploy Promtail

Role: Forward all logging from/in containers to Loki

promtail.yaml

c:\k8s\promtail.yaml

config:
# publish data to loki
  clients:
    - url: http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push
      tenant_id: 1

Install

kubectl create namespace promtail
helm upgrade --values promtail.yaml --install promtail grafana/promtail --namespace promtail

Sample Loki Queries

In a later post I will elaborate on querying but for now I’ll show you two basic queries:

Open Grafana, and start “Explore”, Set to code.

/// Query all Kubernetes events 
{source="kubernetes-event-exporter"} |= ``

/// Query all logs that contain "error", for all running containers
{container=~".+"} |= "error"

Removing and Uninstalling

The commands to remove or uninstalling the various components are:

Uninstall components

helm uninstall event-exporter
helm uninstall loki
helm uninstall promtail

Delete the Kubernetes Cluster

kind delete cluster <name of your cluster>

Leave a Comment