Evaluating the Effect of Distributional Assumptions on Statistical Hypothesis Testing and Model Interpretability

Posted: Mar 05, 2012

• Aiden Carter
• Aiden Clark
• Aiden King

Abstract

Statistical hypothesis testing represents one of the most widely employed methodologies across scientific disciplines, providing a formal framework for drawing inferences from data. The theoretical foundations of these tests, developed primarily in the early 20th century, rest upon specific assumptions about the underlying data distributions. Traditional statistical education emphasizes the importance of verifying assumptions such as normality, independence, and homoscedasticity before applying parametric tests. However, in contemporary data science practice, these verification steps are often overlooked or treated as mere formalities, particularly as computational power has enabled the application of statistical methods to increasingly complex and high-dimensional datasets. The proliferation of machine learning and its emphasis on predictive performance has further marginalized concerns about distributional assumptions, creating a troubling disconnect between statistical theory and applied practice. This research addresses this gap by systematically examining how violations of distributional assumptions affect not only the validity of statistical conclusions but also the interpretability of resulting models. While previous work has primarily focused on Type I and Type II error rates under distributional violations, our investigation extends to the less-explored territory of how these violations propagate through the inference pipeline to affect feature importance, confidence estimates, and overall model interpretability. Our research is motivated by three fundamental questions that remain inadequately addressed in the literature: To what extent do realistic distributional violations common in modern datasets impact the reliability of standard statistical tests? How do these violations systematically bias interpretability metrics commonly used in explanatory modeling? Can we develop a unified framework for quantifying and mitigating the sensitivity of statistical inference to distributional assumptions? By addressing these questions, we aim to bridge the gap between theoretical statistics and applied data science, providing practitioners with actionable insights for more robust statistical practice.