Skip to content

Latest commit

 

History

History
164 lines (143 loc) · 4.93 KB

TUTORIAL.md

File metadata and controls

164 lines (143 loc) · 4.93 KB

kaae

Proof-of-Concept Tutorial

This tutorial will illustrate a working example of kaae for alerting

WARNING: This guide is a work-in-progress and should not be used in production!

Requirements

  • Elasticsearch 2.3.x
  • Kibana 4.5+
  • shell + curl to execute commands

Setup

Before starting, download and install the latest dev version of the plugin:

git clone https://github.com/chenryn/kaae
cd kaae && npm install && npm run package
/opt/kibana/bin/kibana plugin --install kaae -u file://`pwd`/kaae-latest.tar.gz

Dataset

To illustrate the logic and elements involved with kaae we will generate some random data and insert it to Elasticsearch. Our sample JSON object will report a UTC @timestamp and mos value per each interval:

{"mos":4,"@timestamp":"2016-07-17T15:56:02.890"}

The following BASH script will produce our entries for a realistic example:

#!/bin/bash
INDEX=`date +"%Y.%m.%d"`
SERVER="http://127.0.0.1:9200/mos-$INDEX/mos/"

echo "Press [CTRL+C] to stop.."
while :
do
	header="Content-Type: application/json"
	timestamp=`TZ=UTC date +"%Y-%m-%dT%T.%3N"`
	mos=$(( ( RANDOM % 5 )  + 1 ))
	mystring="{\"mos\":${mos},\"@timestamp\":\"${timestamp}\"}"
	echo $mystring;
	curl -sS -i -XPOST -H "$header" -d "$mystring" "$SERVER"
	sleep 5
done
  • Save the file as elasticgen.sh and execute it for a few minutes

Watcher rule

To illustrate the trigger logic, we will create an alert for an aggregation against the data we just created.

The basic example will use simple parameters:

  • Run each 60 seconds
  • Target the daily mos-* index with query aggregation
  • Trip condition when aggregations.avg.value < 3
  • Email action with details
curl -XPUT http://127.0.0.1:9200/watcher/watch/mos -d'
{
  "trigger": {
    "schedule" : { "later" : "every 60 seconds"  }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "<mos-{now/d}>", "<mos-{now/d-1d}>"  ],
        "body" : {
          "query" : {
            "filtered" : {
              "query": {
		"query_string": {
		  "query": "mos:*",
		  "analyze_wildcard": true
		}
	      },
              "filter" : { "range" : { "@timestamp" : { "from" : "now-5m"  } } }
            }
          },
           "aggs": {
		"avg": {
		  "avg": {
		    "field": "mos"
		  }
		}
	  }
        }
      }
    }
  },
  "condition" : {
    "script" : {
      "script" : "payload.aggregations.avg.value < 3"
    }
  },
  "transform" : {
    "search" : {
      "request" : {
        "indices" : [ "<mos-{now/d}>", "<mos-{now/d-1d}>"  ],
        "body" : {
          "query" : {
            "filtered" : {
              "query": {
		"query_string": {
		  "query": "mos:*",
		  "analyze_wildcard": true
		}
	      },
              "filter" : { "range" : { "@timestamp" : { "from" : "now-5m"  } } }
            }
          },
           "aggs": {
		"avg": {
		  "avg": {
		    "field": "mos"
		  }
		}
	  }
        }
      }
    }
  },
  "actions" : {
    "email_admin" : {
    "throttle_period" : "15m",
    "email" : {
      "to" : "[email protected]",
      "subject" : "Low MOS Detected: {{payload.aggregations.avg.value}} ",
      "priority" : "high",
      "body" : "Low MOS Detected:\n {{payload.aggregations.avg.value}} average with {{payload.aggregations.count.value}} measurements in 5 minutes"
    }
    }
  }
}'

Extending Logic

The basic Watcher can be extended and improved following the same logic used with the stock _Watcher, for example by using transform to insert detections back in ES. An interesting set of examples is available here

Alarm Triggering

Kibana/Kaae will automatically fetch and schedule jobs, executing the watcher queries according to the trigger.schedule parameter, validating their results according to the provided condition.script

Check output

Assuming all data and scripts are correctly executed, you should start seeing output in your kibana logs.

Positive Match (Kibana/UI)

Positive Match (console)
Jul 17 17:55:00 es2pcap kibana[44702]: KAAE Payload: { took: 21,
Jul 17 17:55:00 es2pcap kibana[44702]: timed_out: false,
Jul 17 17:55:00 es2pcap kibana[44702]: _shards: { total: 186, successful: 186, failed: 0 },
Jul 17 17:55:00 es2pcap kibana[44702]: aggregations: { avg: { value: 2.9069767441860463 } } }
Jul 17 17:55:00 es2pcap kibana[44702]: KAAE Condition: payload.aggregations.avg.value < 3
Jul 17 17:55:00 es2pcap kibana[44702]: Low MOS Detected: 2.9069767441860463  Low MOS Detected:
Jul 17 17:55:00 es2pcap kibana[44702]: 2.9069767441860463 average with  measurements in 5 minutes

To be continued ...