An Optimized Stacking Ensemble of Deep Learning and Machine Learning Models with Boruta-Based Feature Selection for Android Malware Detection
DOI:
https://doi.org/10.62411/faith.3048-3719-356Keywords:
Android Malware, Boruta, Class Weighting, Deep Learning, Ensemble Learning, Feature Selection, Machine Learning, Malware DetectionAbstract
Malware, particularly within the Android ecosystem, continues to evolve in complexity, rendering traditional signature-based detection techniques ineffective against new and obfuscated threats. This study addresses the challenge of achieving highly accurate and robust malware detection in the presence of high-dimensional and potentially noisy feature spaces. The objective is to develop a feature-optimized hybrid framework that leverages the complementary strengths of deep learning and machine learning models. To this end, a stacking-based hybrid model integrating TabNet, Deep Neural Network, and XGBoost as base learners, with Logistic Regression as the meta-learner, is proposed. The methodology incorporates Boruta-based feature selection to eliminate irrelevant and redundant features, thereby improving model generalization, efficiency, and robustness. Rather than relying on explicit data resampling or class balancing techniques, the proposed approach enhances detection stability under naturally imbalanced data distributions through feature optimization and heterogeneous ensemble learning. The model is evaluated on two benchmark datasets, TUANDROMD and CIC-AndMal2017, to ensure robustness and generalizability. Experimental results demonstrate that on the CIC-AndMal2017 dataset, the hybrid model achieved an accuracy of 0.978945, precision of 0.969766, recall of 0.970508, F1-score of 0.970137, and AUROC of 0.996030, outperforming all individual base models. Similarly, on the TUANDROMD dataset, the hybrid framework attained superior performance with an accuracy of 0.997760, precision of 0.997203, recall of 1.000000, F1-score of 0.998599, and AUROC of 0.999423, again surpassing the base learners. These results confirm that the proposed feature-optimized ensemble stacking approach improves malware detection performance and provides a robust and scalable solution for real-world cybersecurity applications.
Downloads
References
D. Elagina, “Number of smartphone users worldwide from 2016 to 2029,” Statista, 2026. https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/
Statista Research Department, “Global market share held by mobile operating systems from 2009 to 2025,” Statista, 2026. https://www.statista.com/statistics/272698/global-market-share-held-by-mobile-operating-systems-since-2009/
M. Benmalek, “Ransomware on cyber-physical systems: Taxonomies, case studies, security gaps, and open challenges,” Internet Things Cyber-Physical Syst., vol. 4, pp. 186–202, 2024, doi: 10.1016/j.iotcps.2023.12.001.
M. Cen, F. Jiang, X. Qin, Q. Jiang, and R. Doss, “Ransomware early detection: A survey,” Comput. Networks, vol. 239, p. 110138, Feb. 2024, doi: 10.1016/j.comnet.2023.110138.
M. Baykara and E. Colak, “A review of cloned mobile malware applications for android devices,” in 2018 6th International Symposium on Digital Forensic and Security (ISDFS), Mar. 2018, pp. 1–5. doi: 10.1109/ISDFS.2018.8355388.
M. Conti, T. Dargahi, and A. Dehghantanha, “Cyber Threat Intelligence: Challenges and Opportunities,” in Advances in Information Security, 2018, pp. 1–6. doi: 10.1007/978-3-319-73951-9_1.
A. Pagan and K. Elleithy, “A Multi-Layered Defense Approach to Safeguard Against Ransomware,” in 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC), Jan. 2021, pp. 0942–0947. doi: 10.1109/CCWC51732.2021.9375988.
B. Yamany, M. S. Elsayed, A. D. Jurcut, N. Abdelbaki, and M. A. Azer, “A Holistic Approach to Ransomware Classification: Leveraging Static and Dynamic Analysis with Visualization,” Information, vol. 15, no. 1, p. 46, Jan. 2024, doi: 10.3390/info15010046.
M. M. Raza et al., “Malware Detection and AI Integration: A Systematic Review of Current Trends and Future Directions,” Comput. Model. Eng. Sci., vol. 146, no. 3, pp. 1–10, 2026, doi: 10.32604/cmes.2025.074164.
V. Kouliaridis and G. Kambourakis, “A Comprehensive Survey on Machine Learning Techniques for Android Malware Detection,” Information, vol. 12, no. 5, p. 185, Apr. 2021, doi: 10.3390/info12050185.
D. Gibert, C. Mateu, and J. Planes, “The rise of machine learning for detection and classification of malware: Research developments, trends and challenges,” J. Netw. Comput. Appl., vol. 153, p. 102526, Mar. 2020, doi: 10.1016/j.jnca.2019.102526.
M. A. Rahman, G. A. Francia, and H. Shahriar, “Leveraging GANs for Synthetic Data Generation to Improve Intrusion Detection Systems,” J. Futur. Artif. Intell. Technol., vol. 1, no. 4, pp. 429–439, Feb. 2025, doi: 10.62411/faith.3048-3719-52.
M. S. Masari, M. A. Danladi, I. L. Onyinye, and L. K. Tohomdet, “Android Malware Detection Using Machine Learning with SMOTE-Tomek Data Balancing,” J. Comput. Theor. Appl., vol. 3, no. 3, pp. 302–313, Jan. 2026, doi: 10.62411/jcta.15084.
A. K. Jilani and S. Samreen, “Ensemble Based Android Malware Detection Using a Bio-Inspired Meta-Heuristic Feature Engineering Approach,” in 2024 International Conference on Computing, Internet of Things and Microwave Systems (ICCIMS), Jul. 2024, pp. 1–6. doi: 10.1109/ICCIMS61672.2024.10690649.
M. Ibadullah, S. A. Amalina, W. Ghozi, and F. A. Rafrastara, “Machine Learning-based Malware Detection on Android Operating System using AdaBoost Algorithm and ReliefF Feature Selection Method,” in 2024 International Seminar on Application for Technology of Information and Communication (iSemantic), Sep. 2024, pp. 359–364. doi: 10.1109/iSemantic63362.2024.10762096.
Y. Pristyanto, A. F. Nugraha, B. Wulansari, M. Sulistiyono, and A. Sunyoto, “Enhancement of Machine Learning Models Using Intersection Filtering Model Based on Recursive Feature Elimination on Specified Android Malware Classification,” Int. J. Intell. Eng. Syst., vol. 18, no. 3, pp. 836–852, Apr. 2025, doi: 10.22266/ijies2025.0430.56.
T. Etem, “Comparative Analysis of Principle Component Analysis and Anova Feature Selection in Malware Detection,” Firat Univ. J. Exp. Comput. Eng., vol. 5, no. 1, pp. 299–315, Feb. 2026, doi: 10.62520/fujece.1635121.
A. Fenjan, D. J. Md Desa, D. S. Elaskari, A. Theeb, and A. S. Alsayafi, “Addressing Data Imbalance in Malware Detection: A Comparison Analysis of Machine Learning Models for Malware Detection on Balanced and Imbalanced Datasets,” in Proceedings of the Sixth International Conference on Digital Age & Technological Advances for Sustainable Development, May 2025, pp. 1–5. doi: 10.1145/3747897.3747898.
S. Al Ali et al., “Android Malware Detection Using Machine Learning,” in 2024 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS), Nov. 2024, pp. 79–84. doi: 10.1109/IoTaIS64014.2024.10799339.
E. S. Akkaya and E. V. Altay, “Investigating the Performance of Machine Learning Methods for Malware Detection,” in EAI/Springer Innovations in Communication and Computing, 2025, pp. 329–340. doi: 10.1007/978-3-031-88999-8_25.
H. Abdulla, “Android Malware Detection: A Machine Leaning Approach,” arXiv. Nov. 02, 2025. [Online]. Available: http://arxiv.org/abs/2511.00894
M. K. Shah, “AI-Based Framework for Ransomware Detection in Android Systems: Enhancing Mobile Security,” in 2025 5th International Conference on Artificial Intelligence and Signal Processing (AISP), Nov. 2025, pp. 1–8. doi: 10.1109/AISP68263.2025.11396254.
N. G. Ambekar, N. N. Devi, S. Thokchom, and Yogita, “TabLSTMNet: enhancing android malware classification through integrated attention and explainable AI,” Microsyst. Technol., vol. 31, no. 3, pp. 695–713, Mar. 2025, doi: 10.1007/s00542-024-05615-0.
Haris Mehmood, Muhammad Kamran Abid, Muhammad Fuzail, Ahmad Naeem, and Naeem Aslam, “Enhancing Real-time Android Malware Detection using Deep Learning and Fuzzy Logic-based Hybrid Models,” Kashf J. Multidiscip. Res., vol. 2, no. 06, pp. 161–170, Jun. 2025, doi: 10.71146/kjmr494.
A. K. J, N. Das Patel, S. D, and A. Patel, “Behavioral Malware Detection via API Call Sequences: A Comparative Study of LSTM and Transformer Architectures Using NLP-Inspired Representations,” J. Comput. Theor. Appl., vol. 3, no. 4, pp. 443–456, Apr. 2026, doi: 10.62411/jcta.15811.
T. Kacem and S. Tossou, “Trandroid: An Android Mobile Threat Detection System Using Transformer Neural Networks,” Electronics, vol. 14, no. 6, p. 1230, Mar. 2025, doi: 10.3390/electronics14061230.
M. Arya, S. Arya, and S. Arya, “H2O AutoML for Malware Detection: A Signal Processing Perspective on Accuracy, Efficiency, and Interpretability,” in 2025 IEEE International Conference on Signals and Systems (ICSigSys), Nov. 2025, pp. 77–83. doi: 10.1109/ICSigSys67277.2025.11269166.
M. R. Abdellah et al., “Enhanced Federated Learning Framework Based on Deep Learning and Neutrosophic Set for Android Malware Classification,” Neutrosophic Sets Syst., vol. 82, no. 1, 2025, [Online]. Available: https://digitalrepository.unm.edu/cgi/viewcontent.cgi?article=3189&context=nss_journal
A. Daulay, K. Ramli, R. Harwahyu, T. Hidayat, and B. Pranggono, “Novel Federated Graph Contrastive Learning for IoMT Security: Protecting Data Poisoning and Inference Attacks,” Mathematics, vol. 13, no. 15, p. 2471, Jul. 2025, doi: 10.3390/math13152471.
D. Mughole Kalimumbalo et al., “SecFedMDM-1: A Federated Learning-Based Malware Detection Model for Interconnected Cloud Infrastructures,” IEEE Access, vol. 13, pp. 101246–101261, 2025, doi: 10.1109/ACCESS.2025.3577706.
R. B. Coulibaly, T. F. Bissyandé, R. Kafando, A. Sabané, and A. K. Kabore, “Privacy-Preserving Android Malware Detection Using Deep Federated Learning,” in 2025 Cybersecurity4D (C4D), Aug. 2025, pp. 1–7. doi: 10.1109/C4D65382.2025.11306458.
P. Borah, D. Bhattacharyya, and J. Kalita, “Malware Dataset Generation and Evaluation,” in 2020 IEEE 4th Conference on Information & Communication Technology (CICT), Dec. 2020, pp. 1–6. doi: 10.1109/CICT51604.2020.9312053.
M. S. Haque, M. S. Hossain, R. A. Robin, A. S. Tarisha, S. Ahmmed, and J. N. Mukta, “Android Malware Analysis.” Mendeley Data, 2026. doi: 10.17632/bpkksc9v5s.5.
M. B. Kursa and W. R. Rudnicki, “Feature Selection with the Boruta Package,” J. Stat. Softw., vol. 36, no. 11, 2010, doi: 10.18637/jss.v036.i11.
A. Wajahat et al., “An effective deep learning scheme for android malware detection leveraging performance metrics and computational resources,” Intell. Decis. Technol., vol. 18, no. 1, pp. 33–55, Feb. 2024, doi: 10.3233/IDT-230284.
A. Wajahat et al., “Outsmarting Android Malware with Cutting-Edge Feature Engineering and Machine Learning Techniques,” Comput. Mater. Contin., vol. 79, no. 1, pp. 651–673, 2024, doi: 10.32604/cmc.2024.047530.
T. Palabaş, “Android malware classification using basic machine learning methods,” Adıyaman Üniversitesi Mühendislik Bilim. Derg., vol. 11, no. 23, pp. 190–202, Aug. 2024, doi: 10.54365/adyumbd.1462488.
A. Museeb, Y. Hamed, Y. Baashar, A. M. Jamal Kanaan-Jebna, A. Amazigh Hamza, and R. Sokkalingam, “Android Malware Detection Using API Calls and Permissions With Random Forest Classifier,” IEEE Access, vol. 14, pp. 6464–6480, 2026, doi: 10.1109/ACCESS.2026.3651861.
A. Museeb, Y. Hamed, and H. Louis Tan, “Hybrid Random Forest and XGBoost Approach for Android Malware Detection,” Res. J. Maaref Univ. Appl. Sci., vol. 2, no. 1, p. 6, Feb. 2026, doi: 10.66422/wh0h4a90.
S. Rekik, S. Mehmood, and M. Alkhonaini, “AdaptivePixGuard: Attention-Enhanced Temporal Convolutions for Robust Mobile Pixnapping Detection,” IEEE Access, vol. 14, pp. 34072–34095, 2026, doi: 10.1109/ACCESS.2026.3669496.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Maiauduga Abdullahi Danladi, Maryam Sufiyanu Masari, Ubakaghinwa Paul Chigbu, Abdulrashid Abdulrauf
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
