Do you know: how to discover and monitor your files with Autodiscovery on a Kubernetes cluster?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to discover and monitor your files with Autodiscovery on a Kubernetes cluster.

Background

Elastic Beats includes autodiscovery configuration to monitor volumes within containers, across pods, nodes, and even shared volumes across the cluster. This functionality enables Beats to automatically detect and collect logs and events from newly created pods, containers, and services. The autodiscovery feature of Elastic Beats in Kubernetes proves highly effective for monitoring various levels of volume, ranging from node-level metrics to detailed container and volume-level metrics. This facilitates comprehensive monitoring of a Kubernetes environment with minimal manual configuration required.

Solution

Step 1

Ensure that autodiscovery is enabled in your manifest file with requirement fields.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "myapp"
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log

Step 2

When using Elastic Beats to monitor logs in Kubernetes, the choice of mount propagation can significantly impact how logs are accessed and collected.

Ensure that mount propagation is allowed in the Kubernetes volume configuration.

See the options: mount propagation.

Step 3

It is better to deploy Filebeat as a DaemonSet to ensure that the agent is running on all nodes of the cluster.

Download the reference manifest file from GitHub.

“curl -L -O https://raw.githubusercontent.com/elastic/beats/8.13/deploy/kubernetes/filebeat-kubernetes.yaml”

By default, the agent will operate under the ‘kube-system’ namespace. You can always adjust the configuration to meet your requirements.

Need help with your Elastic challenges? Contact our experts.

With our 25+ Elastic certified consultants, Devoteam is your partner for developing and implementing Monitoring & Observability solutions that facilitate optimal IT control, from Business & IT Operations dashboards to centralized logging and proactive alerting.