Log processing

Overview

Vulture is able to receive logs using SYSLOG protocol, Web API, Redis and Kafka. Thanks to its internal redis cache, Vulture can also receive logs from external Elasticsearch beats such as winlogbeat, filebeat, auditbeat, metricbeat (using a redis output). It can also deals with PCAP network capture and receive netflow stream.

rsyslog and filebeat are used internally by Vulture to achieve these purposes.

Incoming TCP flow can be load-balanced over several Vulture instances, thanks to haproxy.

Logs are JSON-normalized (ECS Schema) and enriched with contextual tags and threat intelligence feeds (Open Source or from the Advens CERT if a professional subscription is purchased).

Then detection rules, powered by machine learning and neural networks, are applied to detect anomalies and trigger alerts.

All theses tasks run in parallel, using high-performance C / C++ threads, with low memory consumption.

Log pipeline

Normalization and enrichment

Logs are parsed in real time via rsyslog / liblognorm and transformed into a JSON structure. Then a field conversion occurs to adopt, as much as possible, an ECS format standard.

It is possible to enrich logs, based on an IP address, a hostname, an URL, a hash or whatever… and to add some “tags” to the log line. It allows security analysts to have more insights on the logs and the eventual security concerns.

Here is a sample of additional data added by Vulture whenever an enrichment rule is triggered :

advens : {
    source {
        ip : {
            properties : {
                "cvss_max": 10,
                "os": "Microsoft Windows 7",
                "network": "EXTERNAL"
            },
            "tags" : [ "VIP", "LAPTOP" ],
            "reputation": [ "Emotet" ],
            "ioc": [ "Nephilim Ransomware" ],
            "geo": {
                "city_name": "Gol Gol Road",
                "continent_name": "Oceania",
                "country_iso_code": "AU",
                "country_name": "Australia",
                "location": {
                    "accuracy_radius": 1000,
                    "latitude": -33.494,
                    "longitude": 143.2104,
                    "time_zone": "Australia/Sydney"
                }
            }
        }
    }
}

Anomaly detection and alerting

Thanks to the YARA engine, and thanks to the “mmdarwin” connector for rsyslog, it is possible to write detection rules that will apply on the log processed by Vulture.

In addition to this “basic engine”, Vulture features a powerful machine learning framework with ready-to-use security use-cases based on the following algorithmes (some use-case and algorithms require an Advens subscription):

• DGAD: Domain Generated Algorithm Detection: A neural network to detect generated domains use by malware
• UNAD: UNsupervised Anomaly Detection
• VAML: Detection of abnormal VAriation using a Machine Learning approach via a specific, trained, model
• VAST: Detection of abnormal VAriation using a STatistic approach
• SOFA: Scan Outlier Finder and Analysis, useful to detect assets “outside the norm”
• UEBA: Users and Entities Behavior Analysis

Output modules

Using the numerous rsyslog output modules, Vulture can forward enriched logs and alerts to any SIEM or databases such as MongoDB or Elasticsearch.