Type I Error Rates are Not Usually Inflated

Rubin, Mark (2024) Type I Error Rates are Not Usually Inflated. [Preprint]

This is the latest version of this item.

Text
Rubin (2024) - Type I Error Rates are Not Usually Inflated.pdf
Download (398kB)

Official URL: https://doi.org/10.36850/4d35-44bd

Abstract

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

Export/Citation:

Social Networking:

Share |

Item Type:

Preprint

Creators:

Creators	Email	ORCID
Rubin, Mark	Mark-Rubin@outlook.com	0000-0002-6483-8561

Keywords:

exploratory analyses; false positives; forking paths; HARKing; model misspecification; multiple comparisons; multiple testing; optional stopping; p-hacking; questionable research practices; replication crisis; selective inference; significance testing; statistical inference; Type I error inflation; Type I error rate inflation; Type I error rates

Subjects:

General Issues > Data
General Issues > Evidence
General Issues > Explanation
General Issues > Models and Idealization
Specific Sciences > Probability/Statistics
Specific Sciences > Psychology

Depositing User:

Dr Mark Rubin

Date Deposited:

16 Nov 2024 13:51

Last Modified:

16 Nov 2024 13:51

Item ID:

24226

Official URL:

https://doi.org/10.36850/4d35-44bd