Each week, a new “Do You Know” will be posted on our Elastic Technical Knowledge Hub to share useful knowledge to improve the observability using Elasticsearch. These topics originate from day-to-day challenges we solved for our clients. A stepwise description helps you to successfully implement solutions on improving the performance of your deployment and get the best monitoring of your applications using dashboards and alerting.
This week I will discuss: how to easily build an Advanced Watcher by first creating a data table to obtain the requested json and use it in the Advanced Watcher’s input.
Background
Just imagine you want to have an alert when an error occurs. In addition to that, you want the transactionID and the associated error message in the output of your alert. You build this by using the Advanced Watcher tool in Kibana. In the input section of the Advanced Watcher you must write the filter settings and aggregation in json. Just to avoid making mistakes in your configuration, it is easier to let Kibana write the json for you!
Solution
Step 1
In Kibana, go to the menu and choose Visualization Library → Create visualization → Aggregation based → Data table. Choose your index and start building the table containing the information you want in the output of the watcher. In this example, add a filter on the top of your screen: Severity.keyword: ERROR. On the right hand side add a bucket and choose Aggregation: Terms and Field: TransactionID. Add a next bucket by clicking on the Add button. Choose Aggregation: Terms and Field: ErrorMessage. And click on Update.
Step 2
To retrieve your request to fill this table with the fields and values we just defined, go to the top right corner and choose: Inspect → View: Data → Request → Request.
You request looks like:
{
"aggs": {
"2": {
"terms": {
"field": "TransactionID.keyword",
"order": {
"_count": "desc"
},
"size": 500
},
"aggs": {
"3": {
"terms": {
"field": "ErrorMessage.keyword",
"order": {
"_count": "desc"
},
"size": 500
}
}
}
}
},
"size": 0,
"fields": [
{
"field": "@timestamp",
"format": "date_time"
}
],
"script_fields": {},
"stored_fields": [
"*"
],
"runtime_mappings": {},
"_source": {
"excludes": []
},
"query": {
"bool": {
"must": [],
"filter": [
{
"match_phrase": {
"Severity.keyword": "ERROR"
}
},
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "now-30m"
}
}
}
],
"should": [],
"must_not": []
}
}
}We are going to use the aggregation and query in the Advanced Watcher.
Step 3
Go to the Kibana menu and navigate to: Stack Management → Watcher → Create → Create Advanced Watch. By default you get the following Advanced Watcher configuration:
{
"trigger": {
"schedule": {
"interval": "30m"
}
},
"input": {
"search": {
"request": {
"body": {
"size": 0,
"query": {
"match_all": {}
}
},
"indices": [
"*"
]
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 10
}
}
},
"actions": {
"my-logging-action": {
"logging": {
"text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
}
}
}
}
Step 4
Now we need to start adding parts of the json into the Advanced Watcher. This concerns the query part and aggregations we obtained from our table.
"query": {
"bool": {
"must": [],
"filter": [
{
"match_phrase": {
"Severity.keyword": "ERROR"
}
},
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "now-30m"
}
}
}
],
"should": [],
"must_not": []
}
}
"aggs": {
"2": {
"terms": {
"field": "TransactionID.keyword",
"order": {
"_count": "desc"
},
"size": 500
},
"aggs": {
"3": {
"terms": {
"field": "ErrorMessage.keyword",
"order": {
"_count": "desc"
},
"size": 500
}
}
}
}
}Step 5
The complete watcher configuration will look like this, after adding these segments into the input part of the watcher:
{
"trigger": {
"schedule": {
"interval": "30m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"*"
],
"rest_total_hits_as_int": true,
"body": {
"query": {
"bool": {
"must": [],
"filter": [
{
"match_phrase": {
"Severity.keyword": "ERROR"
}
},
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "now-30m"
}
}
}
],
"should": [],
"must_not": []
}
},
"aggs": {
"2": {
"terms": {
"field": "TransactionID.keyword",
"order": {
"_count": "desc"
},
"size": 500
},
"aggs": {
"3": {
"terms": {
"field": "ErrorMessage.keyword",
"order": {
"_count": "desc"
},
"size": 500
}
}
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 1
}
}
},
"actions": {
"my-logging-action": {
"logging": {
"text": "TransactionID: {{ctx.payload.aggregations.2.buckets.0.key}}, ErrorMessage: {{ctx.payload.aggregations.2.buckets.0.3.buckets.0.key}}"
}
}
}
}
Need help with your Elastic challenges? Contact our experts.
With our 25+ Elastic certified consultants, Devoteam is your partner for developing and implementing Monitoring & Observability solutions that facilitate optimal IT control, from Business & IT Operations dashboards to centralized logging and proactive alerting.
