Exploring the Role of Randomization Tests in Small Sample Inference and Exact Hypothesis Testing Approaches

Posted: Mar 19, 2018

• Jacob Miller
• Zoey Anderson
• Jacob Nguyen

Abstract

Statistical inference in small-sample settings presents unique challenges that conventional parametric methods often fail to adequately address. The reliance on asymptotic approximations and strict distributional assumptions can lead to inflated Type I error rates and reduced power when sample sizes are limited. Randomization tests, also known as permutation tests, offer an attractive alternative by providing exact control of Type I error rates without requiring large-sample theory or specific distributional assumptions. Despite their theoretical appeal, randomization tests have not been widely adopted in many applied fields, partly due to computational limitations and lack of comprehensive performance evaluations in small-sample scenarios. This paper addresses the gap in understanding how randomization tests perform across various small-sample conditions and develops novel methodological extensions to enhance their practical utility. We investigate the fundamental properties of randomization tests when sample sizes are severely constrained, exploring their behavior under different data generating processes, effect sizes, and experimental designs. Our research questions focus on three main areas: first, how do randomization tests compare to traditional parametric and non-parametric methods in maintaining nominal Type I error rates across diverse small-sample scenarios; second, what are the power characteristics of randomization tests relative to alternative approaches when sample sizes are limited; and third, how can randomization tests be adapted to address complex experimental designs and data structures commonly encountered in practice. Our contributions are both methodological and practical. We develop innovative algorithmic implementations that improve computational efficiency without sacrificing statistical properties, making randomization tests more accessible for routine application. We also provide comprehensive performance evaluations across a wide range of conditions, offering clear guidance for practitioners facing small-sample inference challenges. The theoretical framework we present extends existing randomization test methodology to handle situations involving multiple testing, complex dependencies, and non-standard experimental designs.