Assessing the Relationship Between Statistical Information Theory and Model Complexity in Predictive Analytics

Posted: Jul 02, 2021

• Olivia Walker
• Jack Smith
• Mason Anderson

Abstract

The intersection of statistical information theory and model complexity represents a fundamental yet underexplored dimension of predictive analytics. Traditional machine learning paradigms have largely approached model complexity through the lens of regularization techniques aimed at preventing overfitting, with methods such as L1/L2 regularization, dropout, and early stopping becoming standard practice. However, these approaches often treat complexity as an adversary to be constrained rather than as a resource to be understood and optimized. This perspective fails to capture the intrinsic informational value that complexity embodies in representing the underlying data-generating processes. In this paper, we propose a paradigm shift in how model complexity is conceptualized and measured. Drawing from foundational principles of statistical information theory, particularly the work of Claude Shannon, and extending these concepts through the lens of algorithmic information theory as developed by Kolmogorov and Chaitin, we develop a novel framework that quantifies complexity not merely as a measure of parameter count or computational requirements, but as an informational resource. Our approach bridges the gap between theoretical information measures and practical machine learning applications, offering new insights into why certain model architectures outperform others and how complexity should be managed in predictive tasks. We address three primary research questions that have received limited attention in existing literature: First, how can we formally quantify the informational content of model complexity beyond traditional metrics? Second, what is the nature of the relationship between information-theoretic complexity measures and predictive performance across different domains? Third, are there fundamental limits to the utility of complexity in prediction tasks that can be derived from information-theoretic principles? These questions challenge conventional wisdom in machine learning and open new avenues for model development and evaluation. Our contributions are both theoretical and practical.