Do you know: how to send a log from one Elasticsearch ingest pipeline through to a second ingest pipeline?

Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.

This week I will discuss: how to send a log from one Elasticsearch ingest pipeline through to a second ingest pipeline.

Background

Logs that are sent to an Elasticsearch ingest pipeline can be sent through to different indices. A log may need to undergo some additional data processing depending on its content and the index it belongs to. Imagine a log is sent to index_pipeline_1 to undergo data processing and is subsequently sent to index_B on a condition. All the logs in index_B have additional data processing required. In order to facilitate that, we must be able to send the log from ingest_pipeline_1 to a dedicated ingest_ pipeline_2 for further data processing.

Solution

Step 1

A log is sent to ingest_pipeline_1 and must be sent to index_B. In order to do that we must write the following statement:

PUT _ingest/pipeline/ingest_pipeline_1

{

 "processors": [
{
      "set": {
        "if": "",
        "ignore_failure": true,
        "field": "_index",
        "value": "index_B"
      }
    }
]
}

Now the log is sent to index_B. The next step is to link the second ingest pipeline to index_B using the index template.

Step 2

We have to create an index template in order to set-up the final ingest_pipeline_2 for logs coming into index_B. This allows us to keep this setting while the index may rollover.

In Kibana, go to Stack Management → Index Management → Index Templates and create a new Index template. Give the template a name and give the index pattern. In our case that is: index_B.

Go to the tab: Index settings and add the second ingest-pipeline as follows:

{
"index": {
      "final_pipeline": "ingest_pipeline_2"
  }
}

Step 3

In case data already exists in your index_B, you need to perform a rollover before the new setting becomes active.

POST index_B/_rollover

Need help with your Elastic challenges? Contact our experts.

With our 25+ Elastic certified consultants, Devoteam is your partner for developing and implementing Monitoring & Observability solutions that facilitate optimal IT control, from Business & IT Operations dashboards to centralized logging and proactive alerting.