Ensemble-Driven Malware Detection with Anomaly Override Capabilities

This paper proposes a hybrid malware detection framework that combines ensemble learning with an anomalybased override mechanism. The ensemble component integrates Random Forest and XGBoost classifiers, while the override mechanism uses Isolation Forest to detect anomalous or uncertain samples. Each model was individually trained and evaluated on a labeled malware dataset, with Random Forest and XGBoost achieving 78.54% and 72.91% accuracy, respectively. Feature selection was performed using mutual information and ANOVA F -score to enhance model performance. In the proposed two-stage decision process, classification outputs from RF and XGBoost are fused through probabilistic averaging, and Isolation Forest acts as an override to flag potential malware. This hybrid strategy effectively reduces false negatives and enhances detection accuracy. The complete framework achieved 91.49% test accuracy and a weighted F 1-score of 0.91, demonstrating its strength in detecting diverse malware types with improved reliability and adaptability.