elastalert.py

SUMMARY

An elastalert rule tester built using python. Tests could be set-up uniquely and could be run by batch. Specified logs are indexed using Elasticsearch 7.4.0 and are used with a custom elastalert alerter. This program covers testing for single matches and log aggregation with field mapping capabilities. (source code)

ENVIRONMENT

ELASTICSEARCH

v7.4.0

VIRTUAL ENVIRONMENT

Python 3.6

OPERATING SYSTEM

Linux (in this case: Ubuntu 18.04)

IMPORTANT NOTES

  • This probably won't work on Windows Machines.

  • The program manually creates a basic config file if none is specified.

  • The program manually sets the rule's alert to elastalerter.alerter.Alert.

  • If mappings are not specified during aggregation, key fields are automatically set as keyword fields:

    query_key: field_name.keyword
metric_agg_key: field_name.keyword

  • Logs to be used in a specific test could be a list of files or a directory containing all logs to be indexed.

    "log": ["log1.json", "log2.json", ...]

 or

"log": "/dir"

  • If a directory is specified, the program will check if it exists as is, in the directory specified in --logs, or in the current working directory.

  • Test results are laid out as follows in results.json:

    {
    "total": 6,
    "pass": 3,
    "fail": 3,
    "tests": {
        "test_2": {
            "result": "PASSED",
            "message": []
        },
        "test_2": {
            "result": "PASSED",
            "message": []
        },
        "test_3": {
            "result": "FAILED",
            "message": [
                "1 LOG(S) MATCHED: log001.json"
            ]
        },
        "test_4": {
            "result": "FAILED",
            "message": [
                "7 LOG(S) DID NOT MATCH: agg_log001.json, agg_log002.json, agg_log003.json, agg_log004.json, agg_log005.json, log001.json, log003.json"
            ]
        },
        "test_agg_1": {
            "result": "FAILED",
            "message": [
                "HITS (5) EXCEEDED THE THRESHOLD"
            ]
        }
    }
}

SET-UP

  1. Download and run elasticsearch 7.4.0 for Linux.

    $ tar xvf elasticsearch-7.4.0-linux-x86_64.tar.gz

  ...

$ elasticsearch-7.4.0/bin/elasticsearch

  ...

  2. Set-up a python virtual environment:

    $ pip3 install virtualenv

$ python3 -m virtualenv v_env

  Using base prefix '/usr'
  New python executable in CURRENT_WORKING_DIRECTORY/v_env/bin/python3
  Also creating executable in CURRENT_WORKING_DIRECTORY/v_env/bin/python
  Installing setuptools, pip, wheel...
  done.

$ source v_env/bin/activate

(v_env) $

  3. Download and install dependencies for elastalerter.py

    download and unzip the files

    (v_env) $ wget https://github.com/jebidiah-anthony/elastalerter/blob/master/elastalerter.zip?raw=true

  ...omitted...
  HTTP request sent, awaiting response... 200 OK
  Length: 4248 (4.1K) [application/zip]
  Saving to: ‘elastalerter.zip’

  elastalerter.zip         100%[================================>]   4.15K  --.-KB/s    in 0s
  ...omitted...

(v_env) $ unzip elastalerter.py 

  Archive:  elastalerter.zip
    inflating: elastalerter.py
    inflating: requirements.txt
    inflating: setup.py

    install dependencies

    (v_env) $ pip install --requirement requirements.txt

  ...omitted...
  Installing collected packages: pytz, tzlocal, six, APScheduler, urllib3, idna, certifi, chardet, requests, 
  aws-requests-auth, blist, docutils, jmespath, python-dateutil, botocore, s3transfer, boto3, pycparser, cffi, 
  configparser, croniter, defusedxml, docopt, jsonschema, mock, PyStaticConfiguration, stomp.py, exotel, 
  envparse, python-magic, pbr, oauthlib, requests-oauthlib, requests-toolbelt, jira, PyYAML, PySocks, PyJWT, 
  twilio, texttable, elasticsearch, future, thehive4py, elastalert, tabulate
  ...omitted...

    install the alerter

    (v_env) $ pip install .

  ...omitted...
  Successfully built elastalerter
  Installing collected packages: elastalerter
  Successfully installed elastalerter-0.0.2

EXECUTION (w/ sample output)

help (-h, --help)

(v_env) $ python elastalerter.py -h
usage: python ./elastalerter.py --logs LOGS_DIR --rules RULES_DIR --expected expected.json

    SAMPLE "expected.json"
    ------------------------------------------------
    {
        "test_1": {
            "rule": "rule.yaml",
            "log": ["log1.json", "log2.json", ...],
            "match": (true | false),
            "enabled": (true | false)
        },
        "test_2": {
            ...
            "log": "/dir"
            ...
        },
        ...
    }

optional arguments:
  -h, --help       show this help message and exit
  --host HOST      elasticsearch instance host address (default: 127.0.0.1)
  --port PORT      elasticsearch instance port (default: 9200)
  --config YAML    elastalert config file to use.
  --mappings JSON  field data type mappings.
  --verbose        output other details

required arguments:
  --expected JSON  JSON outline of test names and expected results
  --logs DIR       directory of the logs to be indexed
  --rules DIR      directory of the rules to run with elastalert

default output

(v_env) $ python elastalerter.py --logs ./logs --rules ./rules --expected expected.json
[+] RUNNING test_1 :
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] TEST ( test_1 ) PASSED

[+] RUNNING test_2 :
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] TEST ( test_2 ) PASSED

[+] RUNNING test_3 :
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] TEST ( test_3 ) FAILED

[+] RUNNING test_4 :
    [+] SPECIFIED LOGS IS A DIRECTORY ( ./logs )
        [ERROR] "test_file" IS NOT A VALID JSON FILE.
        [ERROR] "test_file.json" IS NOT A VALID LOG FILE.
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] TEST ( test_4 ) FAILED

[+] RUNNING test_agg_1 :
    [+] TESTING RULE -- AGG_RULE01 ( ./agg_rule01.yaml )
    [+] TEST ( test_agg_1 ) FAILED

[+] RUNNING test_agg_02 :
    [+] TEST ( test_agg_2 ) WAS DISABLED


[+] RESULTS WERE OUTPUT TO CURRENT_WORKING_DIRECTORY/results.json


    ╒════════════╤══════════╤═════════════════════════════════════════════════════════════╕
    │ TestID     │ RESULT   │ MESSAGE                                                     │
    ╞════════════╪══════════╪═════════════════════════════════════════════════════════════╡
    │ test_1     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_2     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_3     │ FAILED   │ > 1 LOG(S) MATCHED: log001.json                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_4     │ FAILED   │ > 7 LOG(S) DID NOT MATCH: agg_log001.json, agg_log002.json, │
    │            │          │   agg_log003.json, agg_log004.json, agg_log005.json,        │
    │            │          │   log001.json, log003.json                                  │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_agg_1 │ FAILED   │ > HITS (5) EXCEEDED THE THRESHOLD                           │
    ╘════════════╧══════════╧═════════════════════════════════════════════════════════════╛

mappings (--mappings)

> sample mappings.json

{
    "mappings": {
        "properties": {
            "ip": { "type": "ip" },
            "port": { "type": "integer" }
        }
    }
}

> execution

(v_env) $ python elastalerter.py --logs ./logs --rules ./rules --expected expected.json --mappings mappings.json
[+] RUNNING test_1 :
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] TEST ( test_1 ) PASSED
[+] RUNNING test_2 :
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] TEST ( test_2 ) PASSED

[+] RUNNING test_3 :
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] TEST ( test_3 ) FAILED

[+] RUNNING test_4 :
    [+] SPECIFIED LOGS IS A DIRECTORY ( ./logs )
        [ERROR] "test_file" IS NOT A VALID JSON FILE.
        [ERROR] "test_file.json" IS NOT A VALID LOG FILE.
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] TEST ( test_4 ) FAILED

[+] RUNNING test_agg_1 :
    [+] TESTING RULE -- AGG_RULE01 ( ./agg_rule01.yaml )
    [+] TEST ( test_agg_1 ) FAILED

[+] RUNNING test_agg_02 :
    [+] TEST ( test_agg_2 ) WAS DISABLED


[+] RESULTS WERE OUTPUT TO CURRENT_WORKING_DIRECTORY/results.json


    ╒════════════╤══════════╤═════════════════════════════════════════════════════════════╕
    │ TestID     │ RESULT   │ MESSAGE                                                     │
    ╞════════════╪══════════╪═════════════════════════════════════════════════════════════╡
    │ test_1     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_2     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_3     │ FAILED   │ > 1 LOG(S) MATCHED: log001.json                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_4     │ FAILED   │ > 7 LOG(S) DID NOT MATCH: agg_log001.json, agg_log002.json, │
    │            │          │   agg_log003.json, agg_log004.json, agg_log005.json,        │
    │            │          │   log001.json, log003.json                                  │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_agg_1 │ FAILED   │ > HITS (5) EXCEEDED THE THRESHOLD                           │
    ╘════════════╧══════════╧═════════════════════════════════════════════════════════════╛

verbose (--verbose)

(v_env) $ python elastalerter.py --logs ./logs --rules ./rules --expected ./expected.json --verbose
[+] RUNNING test_1 :
    [+] A NEW INDEX ( test_1 ) WAS CREATED
    [+] TIMESTAMP RANGE -- [2019-09-16T15:59:57 - 2019-09-16T15:59:57]
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] UPDATING RESULTS...
    [+] DELETING TEST INDEX...
    [+] TEST ( test_1 ) PASSED

[+] RUNNING test_2 :
    [+] A NEW INDEX ( test_2 ) WAS CREATED
    [+] TIMESTAMP RANGE -- [2019-09-16T15:59:57 - 2019-09-16T15:59:57]
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] UPDATING RESULTS...
    [+] DELETING TEST INDEX...
    [+] TEST ( test_2 ) PASSED

[+] RUNNING test_3 :
    [+] A NEW INDEX ( test_3 ) WAS CREATED
    [+] TIMESTAMP RANGE -- [2019-09-16T15:59:57 - 2019-09-16T15:59:57]
    [+] TESTING RULE -- RULE01 ( ./rule01.yaml )
    [+] UPDATING RESULTS...
    [+] DELETING TEST INDEX...
    [+] TEST ( test_3 ) FAILED

[+] RUNNING test_4 :
    [+] A NEW INDEX ( test_4 ) WAS CREATED
    [+] SPECIFIED LOGS IS A DIRECTORY ( ./logs )
        [ERROR] "test_file" IS NOT A VALID JSON FILE.
        [ERROR] "test_file.json" IS NOT A VALID LOG FILE.
    [+] TIMESTAMP RANGE -- [2019-09-16T15:59:57 - 2019-09-16T15:59:57]
    [+] TESTING RULE -- RULE02 ( ./rule02.yaml )
    [+] UPDATING RESULTS...
    [+] DELETING TEST INDEX...
    [+] TEST ( test_4 ) FAILED

[+] RUNNING test_agg_1 :
    [+] A NEW INDEX ( test_agg_1 ) WAS CREATED
    [+] TIMESTAMP RANGE -- [2019-09-16T15:59:57 - 2019-09-16T15:59:57]
    [+] TESTING RULE -- AGG_RULE01 ( ./agg_rule01.yaml )
    [+] UPDATING RESULTS...
    [+] DELETING TEST INDEX...
    [+] TEST ( test_agg_1 ) FAILED

[+] RUNNING test_agg_02 :
    [+] TEST ( test_agg_2 ) WAS DISABLED


[+] RESULTS WERE OUTPUT TO CURRENT_WORKING_DIRECTORY/results.json

    ╒════════════╤══════════╤═════════════════════════════════════════════════════════════╕
    │ TestID     │ RESULT   │ MESSAGE                                                     │
    ╞════════════╪══════════╪═════════════════════════════════════════════════════════════╡
    │ test_1     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_2     │ PASSED   │                                                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_3     │ FAILED   │ > 1 LOG(S) MATCHED: log001.json                             │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_4     │ FAILED   │ > 7 LOG(S) DID NOT MATCH: agg_log001.json, agg_log002.json, │
    │            │          │   agg_log003.json, agg_log004.json, agg_log005.json,        │
    │            │          │   log001.json, log003.json                                  │
    ├────────────┼──────────┼─────────────────────────────────────────────────────────────┤
    │ test_agg_1 │ FAILED   │ > HITS (5) EXCEEDED THE THRESHOLD                           │
    ╘════════════╧══════════╧═════════════════════════════════════════════════════════════╛
PreviousBlue TeamNextWindows Event Fowarder

Last updated