Evaluating the Sensitivity of Machine Learning Classifiers to Cross-Validation Strategies in Multi-Domain Scientific Abstract Classification
DOI:
https://doi.org/10.62411/faith.3048-3719-440Keywords:
Adaptive cross-validation, Artificial neural network, Cross-validation, Machine learning, Natural language processing, Random forest, Scientific abstract classification, Support vector machineAbstract
The growing volume of scholarly publications across research areas has increased the need for accurate and reliable scientific abstract classification. However, classifier performance may depend not only on the learning algorithm but also on the validation strategy used to estimate performance, assess stability, and determine computational cost. This study evaluates Standard 5-Fold Cross-Validation, Repeated 5-Fold Cross-Validation, and Adaptive 5-Fold Cross-Validation for Random Forest (RF), Support Vector Machine (SVM), and Artificial Neural Network (ANN) classifiers using 629 scientific abstracts from Accounting, Computer Science, and Engineering. The abstracts were represented using a binary document-term matrix (DTM), with vocabulary construction and sparsity filtering (threshold = 0.97) fitted within each training fold to reduce information leakage. A fixed partition yielded 441 modeling observations and 188 held-out test observations (random seed = 100). The results show that the range of mean cross-validation accuracy across strategies was 0.30 percentage points for RF, 0.53 for SVM, and 0.23 for ANN. On the fixed test set, RF accuracy ranged from 96.81% to 97.34%, whereas SVM and ANN maintained accuracies of 96.28% and 97.87%, respectively, across all three strategies. Thus, ANN achieved the highest fixed-test accuracy, while the apparent stability of classifier performance depended on the evaluation measure: ANN showed the smallest variation in mean cross-validation accuracy, whereas SVM and ANN had unchanged fixed-test accuracy across strategies. Repeated 5-Fold Cross-Validation required the longest runtime for all three classifiers, while Adaptive 5-Fold Cross-Validation was fastest for RF and ANN; Standard 5-Fold Cross-Validation was fastest for SVM. These findings indicate that validation strategy affects resampling estimates and computational cost, although its influence on fixed-test accuracy varies by classifier. The results also highlight the importance of distinguishing resampling-based performance from held-out test performance when comparing classifiers.
Downloads
References
W. M. To and B. T. W. Yu, “Rise in higher education researchers and academic publications,” Emerald Open Res., vol. 1, no. 3, Dec. 2023, doi: 10.1108/EOR-03-2023-0008.
N. Ibrahim Altmami and M. El Bachir Menai, “Automatic summarization of scientific articles: A survey,” J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 4, pp. 1011–1028, Apr. 2022, doi: 10.1016/j.jksuci.2020.04.020.
R. M S, R. Kumar V, S. M, and C. A R, “Current Approaches in Abstractive Text Summarization: A Comprehensive Survey and Analysis,” in 2025 International Conference on Artificial Intelligence and Data Engineering (AIDE), Feb. 2025, pp. 840–848. doi: 10.1109/AIDE64228.2025.10987294.
D. A. Q. Moore, O. Yaqub, and B. N. Sampat, “Manual versus machine: How accurately does the Medical Text Indexer (MTI) classify different document types into disease areas?,” PLoS One, vol. 19, no. 3, p. e0297526, Mar. 2024, doi: 10.1371/journal.pone.0297526.
T. Thumbumrung, A. Vasconcelos, and A. Cox, “Complexity and evolution of knowledge boundaries in an interdisciplinary research project,” Knowl. Process Manag., vol. 29, no. 3, pp. 296–306, Jul. 2022, doi: 10.1002/kpm.1727.
G. Pech, C. Delgado, and S. P. Sorella, “Classifying papers into subfields using Abstracts, Titles, Keywords and KeyWords Plus through pattern detection and optimization procedures: An application in Physics,” J. Assoc. Inf. Sci. Technol., vol. 73, no. 11, pp. 1513–1528, Nov. 2022, doi: 10.1002/asi.24655.
H. Allam, L. Makubvure, B. Gyamfi, K. N. Graham, and K. Akinwolere, “Text Classification: How Machine Learning Is Revolutionizing Text Categorization,” Information, vol. 16, no. 2, p. 130, Feb. 2025, doi: 10.3390/info16020130.
V. Gupta, J. D. H. Paul, I. Schmitt, and F. Pyczak, “SciLitMiner: An Intelligent System for Scientific Literature Mining and Knowledge Discovery,” Adv. Intell. Syst., vol. 8, no. 3, Mar. 2026, doi: 10.1002/aisy.202501235.
M. Lan, “Natural Language Processing and Text Mining,” in Artificial Intelligence for Drug Design, Singapore: Springer Nature Singapore, 2026, pp. 189–215. doi: 10.1007/978-981-95-2525-6_18.
A. Palanivinayagam, C. Z. El-Bayeh, and R. Damaševičius, “Twenty Years of Machine-Learning-Based Text Classification: A Systematic Review,” Algorithms, vol. 16, no. 5, p. 236, Apr. 2023, doi: 10.3390/a16050236.
A. Desai, M. Abdelhamid, and N. R. Padalkar, “What is reproducibility in artificial intelligence and machine learning research?,” AI Mag., vol. 46, no. 2, Jun. 2025, doi: 10.1002/aaai.70004.
S. Bates, T. Hastie, and R. Tibshirani, “Cross-Validation: What Does It Estimate and How Well Does It Do It?,” J. Am. Stat. Assoc., vol. 119, no. 546, pp. 1434–1445, Apr. 2024, doi: 10.1080/01621459.2023.2197686.
M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models,” Nat. Commun., vol. 15, no. 1, p. 1829, Feb. 2024, doi: 10.1038/s41467-024-46150-w.
A. Bahmani, “Fusion of Statistical and Stylistic Text Features with SVM for Persian Sentiment Analysis,” J. Futur. Artif. Intell. Technol., vol. 2, no. 4, pp. 534–548, Dec. 2025, doi: 10.62411/faith.3048-3719-287.
Y. C. Goh, X. Q. Cai, W. Theseira, G. Ko, and K. A. Khor, “Evaluating human versus machine learning performance in classifying research abstracts,” Scientometrics, vol. 125, no. 2, pp. 1197–1212, Nov. 2020, doi: 10.1007/s11192-020-03614-2.
Y. Guo, A. Korhonen, I. Silins, and U. Stenius, “Weakly supervised learning of information structure of scientific abstracts—is it accurate enough to benefit real-world tasks in biomedicine?,” Bioinformatics, vol. 27, no. 22, pp. 3179–3185, Nov. 2011, doi: 10.1093/bioinformatics/btr536.
D. R. I. M. Setiadi, D. Marutho, and N. A. Setiyanto, “Comprehensive Exploration of Machine and Deep Learning Classification Methods for Aspect-Based Sentiment Analysis with Latent Dirichlet Allocation Topic Modeling,” J. Futur. Artif. Intell. Technol., vol. 1, no. 1, pp. 12–22, May 2024, doi: 10.62411/faith.2024-3.
M. Khadhraoui, H. Bellaaj, M. Ben Ammar, H. Hamam, and M. Jmaiel, “Machine Learning Classification Models with SPD/ED Dataset: Comparative Study of Abstract Versus Full Article Approach,” in Lecture Notes in Computer Science, 2020, pp. 348–356. doi: 10.1007/978-3-030-51517-1_31.
T. Ishankulov, G. Danilov, K. Kotik, Y. Orlov, M. Shifrin, and A. Potapov, “The Classification of Scientific Abstracts Using Text Statistical Features,” in Volume 290: MEDINFO 2021: One World, One Health – Global Partnership for Digital Innovation, 2022. doi: 10.3233/SHTI220075.
B. Kandimalla, S. Rohatgi, J. Wu, and C. L. Giles, “Large Scale Subject Category Classification of Scholarly Papers With Deep Attentive Neural Networks,” Front. Res. Metrics Anal., vol. 5, Feb. 2021, doi: 10.3389/frma.2020.600382.
I. Tougui, A. Jilbab, and J. El Mhamdi, “Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications,” Healthc. Inform. Res., vol. 27, no. 3, pp. 189–199, Jul. 2021, doi: 10.4258/hir.2021.27.3.189.
V. Lumumba, D. Kiprotich, M. Mpaine, N. Makena, and M. Kavita, “Comparative Analysis of Cross-Validation Techniques: LOOCV, K-folds Cross-Validation, and Repeated K-folds Cross-Validation in Machine Learning Models,” Am. J. Theor. Appl. Stat., vol. 13, no. 5, pp. 127–137, Oct. 2024, doi: 10.11648/j.ajtas.20241305.13.
A. Omarbekova, A. Nazyrova, G. Bekmanova, Z. Lamasheva, A. Gibadullina, and A. Abdikarim, “An explainable machine learning framework for academic stress classification among university students: a comparative multi-dataset study with ablation analysis and statistical significance testing,” Front. Comput. Sci., vol. 8, Jul. 2026, doi: 10.3389/fcomp.2026.1886274.
R. T. Nakatsu, “Validation of machine learning ridge regression models using Monte Carlo, bootstrap, and variations in cross-validation,” J. Intell. Syst., vol. 32, no. 1, Jul. 2023, doi: 10.1515/jisys-2022-0224.
C. Arhiliuc, R. Guns, W. Daelemans, and T. C. E. Engels, “Journal article classification using abstracts: a comparison of classical and transformer-based machine learning methods,” Scientometrics, vol. 130, no. 1, pp. 313–342, Jan. 2025, doi: 10.1007/s11192-024-05217-7.
S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection,” BMC Bioinformatics, vol. 7, no. 1, p. 91, Dec. 2006, doi: 10.1186/1471-2105-7-91.
M. Bhagat and B. Bakariya, “A Comprehensive Review of Cross-Validation Techniques in Machine Learning,” Int. J. Sci. Technol., vol. 16, no. 1, Jan. 2025, doi: 10.71097/IJSAT.v16.i1.1305.
A. Geroldinger, L. Lusa, M. Nold, and G. Heinze, “Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study,” Diagnostic Progn. Res., vol. 7, no. 1, p. 9, May 2023, doi: 10.1186/s41512-023-00146-0.
P. Koukaras and C. Tjortjis, “Data Preprocessing and Feature Engineering for Data Mining: Techniques, Tools, and Best Practices,” AI, vol. 6, no. 10, p. 257, Oct. 2025, doi: 10.3390/ai6100257.
H. Chen and G. Shih, “Classifying Radiology Abstracts with Deep Learning,” medRxiv. Aug. 08, 2021. doi: 10.1101/2021.08.07.21261750.
O. A. Montesinos López, A. Montesinos López, and J. Crossa, “Overfitting, Model Tuning, and Evaluation of Prediction Performance,” in Multivariate Statistical Machine Learning Methods for Genomic Prediction, Cham: Springer International Publishing, 2022, pp. 109–139. doi: 10.1007/978-3-030-89010-0_4.
T. L. A. Dinh and F. Aires, “Nested leave-two-out cross-validation for the optimal crop yield model selection,” Geosci. Model Dev., vol. 15, no. 9, pp. 3519–3535, May 2022, doi: 10.5194/gmd-15-3519-2022.
D. Abriha, P. K. Srivastava, and S. Szabó, “Smaller is better? Unduly nice accuracy assessments in roof detection using remote sensing data with machine learning and k-fold cross-validation,” Heliyon, vol. 9, no. 3, p. e14045, Mar. 2023, doi: 10.1016/j.heliyon.2023.e14045.
G. Nicora, M. Rios, A. Abu-Hanna, and R. Bellazzi, “Evaluating pointwise reliability of machine learning prediction,” J. Biomed. Inform., vol. 127, p. 103996, Mar. 2022, doi: 10.1016/j.jbi.2022.103996.
F. A. Njoku, “Scientific Abstract,” Kaggle, 2026. https://www.kaggle.com/datasets/njokufelixa/scientific-abstract
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Felix Njoku, Saviour Inyang, Imeh Umoren, Ozuomba Simeon
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
