A Pragmatic Approach to Statistical Testing and Estimation (PASTE)

The p-value has dominated research in education and related fields and a statistically non-significant p-value is quite commonly interpreted as ‘confirming’ the null hypothesis (H0) of ‘equivalence’. This is unfortunate, because p-values are not fit for that purpose. This paper discusses three alternatives to the traditional p-value that unfortunately have remained underused but can provide evidence in favor of ‘equivalence’ relative to ‘non-equivalence’: two one-sided tests (TOST) equivalence testing, Bayesian hypothesis testing, and information criteria. TOST equivalence testing and p-values both rely on concepts of statistical significance testing and can both be done with confidence intervals, but treat H0 and the alternative hypothesis (H1) differently. Bayesian hypothesis testing and the Bayesian credible interval aka posterior interval provide Bayesian alternatives to traditional p-values, TOST equivalence testing, and confidence intervals. However, under conditions outlined in this paper, confidence intervals and posterior intervals may yield very similar interval estimates. Moreover, Bayesian hypothesis testing and information criteria provide fairly easy to use alternatives to statistical significance testing when multiple competing models can be compared. Based on these considerations, this paper outlines a pragmatic approach to statistical testing and estimation (PASTE) for research in education and related fields. In a nutshell, PASTE states that all of the alternatives to p-values discussed in this paper are better than p-values, that confidence intervals and posterior intervals may both provide useful interval estimates, and that Bayesian hypothesis testing and information criteria should be used when the comparison of multiple models is concerned.