Rubin, Mark (2024) Type I error rates are not usually inflated. Journal of Trial and Error, 4 (2). pp. 46-71.

Text Rubin (2024) - Type I Error Rates are Not Usually Inflated.pdf - Published Version Available under License Creative Commons Attribution. Download (358kB)

Abstract

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin by introducing the concept of Type I error rates and distinguishing between statistical errors and theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

---

Export/Citation:

EndNote | BibTeX | Dublin Core | ASCII/Text Citation (Chicago) | HTML Citation | OpenURL

Social Networking:

Share |

---

Item Type:	Published Article or Volume
Creators:	Creators Email ORCID Rubin, Mark Mark-Rubin@outlook.com 0000-0002-6483-8561
Keywords:	exploratory analyses; false positives; forking paths; HARKing; model misspecification; multiple comparisons; multiple testing; optional stopping; p-hacking; questionable research practices; replication crisis; selective inference; significance testing; statistical inference; Type I error inflation; Type I error rate inflation; Type I error rates
Subjects:	General Issues > Data General Issues > Evidence General Issues > Explanation General Issues > Models and Idealization Specific Sciences > Probability/Statistics Specific Sciences > Psychology
Depositing User:	Dr Mark Rubin
Date Deposited:	29 Dec 2024 19:29
Last Modified:	29 Dec 2024 19:29
Item ID:	24453
Journal or Publication Title:	Journal of Trial and Error
Official URL:	https://doi.org/10.36850/4d35-44bd
DOI or Unique Handle:	10.36850/4d35-44bd
Subjects:	General Issues > Data General Issues > Evidence General Issues > Explanation General Issues > Models and Idealization Specific Sciences > Probability/Statistics Specific Sciences > Psychology
Date:	16 November 2024
Page Range:	pp. 46-71
Volume:	4
Number:	2
URI:	https://philsci-archive.pitt.edu/id/eprint/24453

Available Versions of this Item

• Type I error rates are not usually inflated. (deposited 15 Dec 2023 03:03)
  • Type I Error Rates are Not Usually Inflated. (deposited 05 Jan 2024 03:06)
    • Type I Error Rates are Not Usually Inflated. (deposited 16 Nov 2024 13:51)
      • Type I error rates are not usually inflated. (deposited 29 Dec 2024 19:29) [Currently Displayed]

Monthly Views for the past 3 years

Monthly Downloads for the past 3 years

Plum Analytics

Altmetric.com

Actions (login required)

View Item