Anomaly Detection in HDFS Logs using Transformer Models and Machine Learning
Sections:
00:00 Introduction
00:38: Dataset and overview
02:35: Training the ensemble model
09:05: Model Evaluation with varying anomaly ratios
15:08: Deploying on an HDFS cluster in AWS EMR
Code:
https://github.com/hasb73/hdfs-anomaly-det...
Anomaly detection in system log streams is a challenging task due to the volume, throughput, variety, and lack of high-quality labelled data. This research proposes a novel ensemble-based anomaly detection system that addresses the challenges faced by traditional methods while also being usable in real-time scenarios, along with its ability to train on a small subset of the actual log data.
The focus of this research is the block-level operations in Hadoop File System (HDFS) Logs. The core system is a weighted voting ensemble that combines predictions from diverse learning techniques, including Deep Learning (MLPClassifier), Classical Machine Learning (SGDClassifier, DecisionTreeClassifier), and a unique Vector Similarity Detector that uses a vector database for nearest-neighbor comparisons on embeddings
To prove practicality, an anomaly detection service utilizes the trained model and consumes the live log stream in a real-time setup on AWS Elastic Map Reduce (EMR) to detect, cache, store the anomalies and alert a system administrator with visualisations and notications. The outcomes from testing, evaluation, and performance metrics, demonstrate the system’s end-to-end effectiveness while also displaying promising results in precision and accuracy, which are vital in this field