Evaluating the Impact of Model Selection Bias on Cross-Validated Statistical Performance Metrics

Posted: Dec 13, 2025

• Lauren Parker
• Layla Evans
• Leah Morris

Abstract

Cross-validation has become the cornerstone of modern machine learning practice, serving as the primary methodology for both model selection and performance estimation. The widespread adoption of techniques such as k-fold cross-validation reflects their perceived robustness in providing unbiased estimates of model generalization error. However, this research identifies a fundamental flaw in the conventional application of cross-validation when the same procedure is used for both model selection and final performance assessment. The phenomenon we term Model Selection Bias (MSB) represents a systematic distortion that occurs when the selection of the best-performing model from a candidate set is followed by performance estimation using the same data partitioning scheme. The problem emerges from the inherent dependency between the model selection process and the subsequent performance evaluation. When researchers employ cross-validation to compare multiple algorithms or hyperparameter configurations, they naturally select the configuration that demonstrates superior cross-validated performance. This selection process, however, introduces an optimistic bias because the chosen model has effectively been optimized for the specific cross-validation splits used during selection. The performance estimate derived from these same splits therefore represents a best-case scenario rather than a true reflection of generalization capability. This investigation addresses several critical research questions that have received limited attention in the existing literature. First, we seek to quantify the magnitude of MSB across different experimental conditions and performance metrics. Second, we examine how dataset characteristics such as sample size, dimensionality, and noise level influence the severity of this bias. Third, we evaluate