Dealing with multiple tools for capturing application logs from different sources can be a hassle for anyone. In this blog post, we'll dive into the steps required to establish centralized logging with Grafana Loki and Grafana Agent. This solution will allow us to unify the collection of logs from Kubernetes pods, VM services, and Azure PAAS services.

Grafana Loki : It is a highly scalable log aggregation system designed for cloud-native environments

Grafana Agent : It is an observability agent that collects metrics and logs from various application for visualization and analysis in Grafana

Requirement:

  • Kuberentes Cluster (Latest version)
  • Helm
  • Azure PAAS(Eventhub and AKS, IOT, etc)
  • VM with any application
  • Azure storage account(Loki backend)
  • Azure subscription with admin privileges
Step 1: Deploy Grafana loki in Kubernetes(K8s) cluster,

Ensure you have admin permission for the k8s cluster.

Before deploying Grafana Loki in k8s cluster, there are certain changes are required in the configuration.

Note : We are going to use backend as Azure Storage container in loki to store the logs and we will use Loki distributed version.

Execute below helm commands to add the Grafana repository,

helm repo add grafana https://grafana.github.io/helm-charts

helm repo update


Execute below command to export the Grafana Loki and Grafana agent configuration via helm,

helm show values grafana/loki-distributed > loki-values.yaml

helm show values grafana/grafana-agent > grafana-agent-values.yaml

In the loki-values.yaml, update the below configuration to use Azure storage account as backend.

schemaConfig:

    configs:

    - from: "2020-09-07"

      store: boltdb-shipper

      object_store: azure

      schema: v11

      index:

        prefix: index_

        period: 24h

  storageConfig:

    boltdb_shipper:

      shared_store: azure

      active_index_directory: /var/loki/index

      cache_location: /var/loki/cache

      cache_ttl: 1h

    filesystem:

      directory: /var/loki/chunks

    azure:

      account_name: === Azure Storage name ===

      account_key: === Azure Storage access key ===

      container_name: === Container Name ===

      request_timeout: 0


Here is the loki-values.yaml.

https://github.com/DevOpsArts/grafana_loki_agent/blob/main/loki-distributed-values.yaml

Next deploy Grafana loki,

Execute below command to deploy Loki in the k8s cluster,

helm upgrade --install --values loki-distributed-values.yaml loki grafana/loki-distributed -n observability --create-namespace


Verify the pods are up and running by using below command,

kubectl get all -n observability


Now all the pods are up and running.


Step 2: Deploy Grafana agent in K8s cluster,

Deploy Grafana Agent in k8s cluster to export the k8s Pod logs to Loki

Update the grafana agent values before deploying.

Replace the grafana-agent-values.yaml file with loki distributed gateway service endpoint in line number 169 in the below file with your namespace. Currently observability is used.

https://github.com/DevOpsArts/grafana_loki_agent/blob/main/grafana-agent-values.yaml

loki.write "local" {
        endpoint {
          url = "http://loki-loki-distributed-gateway.observability.svc.cluster.local/loki/api/v1/push"
        }
      }

In the grafana-agent-values.yaml, Currently it is added to export k8s pod logs, k8s events, etc.

Next deploy Grafana-Agent using below command,

helm install --values grafana-agent-values.yaml grafana-agent grafana/grafana-agent -n observability


Verify the pods are up and running by using below command,

kubectl get all -n observability

Now you can go to azure storage account which is configured in Grafana Loki and verify the logs are getting updated to the respective container.


Step 3: Deploy Grafana, to view the pod logs

Execute below command to install Grafana in K8s cluster,

helm install grafana grafana/grafana -n observability


Verify the pods are up and running by using below command,

kubectl get all -n observability

use below command to get the password for Grafana,

kubectl get secret --namespace observability grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Step 4:  Access Grafana and view the pod logs

Use port forward or use NodePort or Ingress to access the Grafana Dashboard.

- Configure loki as Datasource in Grafana,

Once login to the Grafana, Go to Home > Connections > Search for "Loki", then select it.


Next give connection url as, 

http://loki-loki-distributed-gateway.monitoring1.svc.cluster.local

then select save and test.


- Next import Loki dashboard,

Go to Grafana Home > Dashboard > select New and Import.

Use this below Grafana template ID "15141" , load it and save it.

https://grafana.com/grafana/dashboards/15141-kubernetes-service-logs/?source=post_page-----d2847d526f9e--------------------------------


Next click the dashboard, You can able to view all the pod logs now.


Step 5:  View the Kubernetes(K8s) Events in Grafana,

To view the Kubernetes events, Go to Grafana Home page > Explore > Select "Datasource name" > Select "Job" and value as "loki.source.kubernetes_events"


In this part, We have covered how to deploy Grafana Loki with Azure Storage as the backend, deployed Grafana Agent to view Kubernetes pod logs, and deployed Grafana dashboard to visualize the pod logs and Kubernetes events.

In the next Part 2, We will explain how to configure Grafana Agent for VM applications and send the logs to the same Grafana Loki.

Part 3 : We will cover how to export Azure PAAS services logs to Grafana Loki


Tags

1 Comments

Post a Comment

Previous Post Next Post