Assessing the Impact of Regularization Techniques on Parameter Shrinkage and Overfitting in High-Dimensional Data

Posted: Jul 05, 2025

• Jack Williams
• Harper Mitchell
• Liam Moore

Abstract

The proliferation of high-dimensional datasets across scientific domains has created unprecedented challenges for statistical modeling and machine learning. In these settings, where the number of features often vastly exceeds the number of observations, traditional statistical methods face fundamental limitations due to the curse of dimensionality. Regularization techniques have emerged as essential tools for addressing these challenges by imposing constraints on model complexity and promoting parameter shrinkage. However, despite extensive theoretical development and practical application, the nuanced relationships between different regularization approaches, their parameter shrinkage characteristics, and their effectiveness in controlling overfitting remain incompletely understood. This research addresses critical gaps in our understanding of how regularization techniques behave in complex high-dimensional environments. While L1 and L2 regularization have been widely adopted, their comparative performance under varying data conditions, particularly in the presence of complex correlation structures and heterogeneous noise patterns, requires systematic investigation. Our study introduces a novel evaluation framework that moves beyond conventional performance metrics to examine the dynamic interplay between parameter shrinkage patterns, feature selection accuracy, and generalization performance. The central research questions guiding this investigation focus on understanding how different regularization techniques affect parameter estimation in high-dimensional settings. Specifically, we examine whether traditional regularization methods achieve optimal balance between bias and variance across diverse data conditions, how parameter shrinkage patterns vary with increasing dimensionality and correlation complexity, and whether adaptive regularization strategies can overcome limitations of standard approaches.