A Hybrid Framework for Robust Anomaly Detection: Integrating Unsupervised and Supervised Learning with Advanced Feature Engineering

Abstract

Finding anomalous data is essential in various applications, from cyber security to healthcare to industrial monitoring. Traditional methods- unsupervised or supervised—are far from straightforward; unsupervised methods are notoriously plagued by high false favorable rates and unclear distinction boundaries, while supervised methods tend to rely on a great deal of labeled data, often in limited supply or highly imbalanced. Indeed, these problems call for a unified approach that takes advantage of the benefits of both paradigms for more robust anomaly detection. In this work, we develop a hybrid outlier detection framework combining several unsupervised anomaly scoring models (Isolation Forest, Local Outlier Factor, and One-Class SVM) and XGBoost and Logistic Regression as a supervised classifier. Instead, we combine the proposed algorithm with advanced feature engineering techniques (e.g., topological space optimization) to extract informative features for our data representation. Our empirical studies of diverse benchmark datasets (Arrhythmia, Cardio, Letter, Mammography, MNIST, Satellite, and Speech) indicate that the hybrid model consistently shows a significant improvement over any single method. Our framework consistently reduces false positives and false negatives and is more precise; recall, F1-score, and ROC-AUC are the highest scores for quantitative comparison. We demonstrate the usefulness of the proposed framework by enabling it to handle high-dimensional, imbalanced datasets while leading to meaningful detection results in real-world settings. Establishes a new state-of-the-art performance in anomaly detection while also supplying an approach that is scalable and versatile for complex data environments and forming a basis from which to build toward future integrated anomaly detection systems.