Evaluating the Performance of Hypothesis Testing Procedures Under Non-Normal Distributional Assumptions

Posted: Mar 18, 2017

• Evelyn Scott
• Evelyn White
• Evelyn Wilson

Abstract

Statistical hypothesis testing represents a cornerstone of scientific inquiry across numerous disciplines, providing a formal framework for drawing inferences from empirical data. The theoretical foundation of many commonly employed testing procedures, including the ubiquitous t-test and analysis of variance (ANOVA), rests upon the critical assumption that the underlying data follow a normal distribution. This normality assumption permeates introductory statistics education and practical applications alike, yet empirical evidence consistently demonstrates that real-world data frequently violate this fundamental premise. The consequences of such violations remain inadequately characterized in the existing literature, particularly with respect to the complex interplay between specific distributional characteristics and statistical performance metrics. Traditional approaches to addressing non-normality have typically involved data transformation techniques or the application of nonparametric alternatives. While these methods offer theoretical protection against certain types of distributional violations, their practical efficacy varies considerably across different contexts and sample sizes. The logarithmic transformation, for instance, effectively addresses right-skewed distributions but may introduce substantial bias when applied to data containing zero or negative values. Similarly, nonparametric methods such as the Mann-Whitney U test or Kruskal-Wallis test sacrifice statistical power when the normality assumption actually holds, creating a persistent tension between robustness and efficiency in statistical practice. This research addresses several critical gaps in the current understanding of hypothesis testing performance under non-normal conditions. First, we develop a comprehensive taxonomy of distributional violations that moves beyond simple characterizations of skewness and kurtosis to incorporate multimodal distributions, mixture models, and distributions with varying tail behavior. Second, we introduce a novel evaluation framework that simultaneously considers multiple performance metrics, including Type I error rate control, statistical power, confidence interval coverage, and effect size estimation accuracy. Third, we systematically compare the performance of traditional parametric tests, transformation-based approaches, and modern resampling methods across a wide spectrum of statistical models, such as mixed effects models or structural equation models, represents an important area for future research.