Rubin, Mark (2023) Type I error rates are not usually inflated. [Preprint]

Preview	Text Rubin (2023) - Type I Error Rates are Not Usually Inflated.pdf Download (365kB) | Preview
	Other (Plain Text Bibliography) bibliography.txt - Bibliography Download (21kB)

Abstract

The inflation of Type I error rates is thought to be one of the causes of the replication crisis. Questionable research practices such as p-hacking are thought to inflate Type I error rates above their nominal level, leading to unexpectedly high levels of false positives in the literature and, consequently, unexpectedly low replication rates. In this article, I offer an alternative view. I argue that questionable and other research practices do not usually inflate relevant Type I error rates. I begin with an introduction to Type I error rates that distinguishes them from theoretical errors. I then illustrate my argument with respect to model misspecification, multiple testing, selective inference, forking paths, exploratory analyses, p-hacking, optional stopping, double dipping, and HARKing. In each case, I demonstrate that relevant Type I error rates are not usually inflated above their nominal level, and in the rare cases that they are, the inflation is easily identified and resolved. I conclude that the replication crisis may be explained, at least in part, by researchers’ misinterpretation of statistical errors and their underestimation of theoretical errors.

---

Export/Citation:

EndNote | BibTeX | Dublin Core | ASCII/Text Citation (Chicago) | HTML Citation | OpenURL

Social Networking:

Share |

---

Item Type:	Preprint
Creators:	Creators Email ORCID Rubin, Mark Mark-Rubin@outlook.com 0000-0002-6483-8561
Keywords:	exploratory analyses; false positives; forking paths; HARKing; model misspecification; multiple comparisons; multiple testing; optional stopping; p-hacking; questionable research practices; replication crisis; selective inference; significance testing; statistical inference; Type I error inflation; Type I error rate inflation; Type I error rates
Subjects:	General Issues > Data General Issues > Evidence General Issues > Explanation General Issues > Models and Idealization Specific Sciences > Probability/Statistics Specific Sciences > Psychology
Depositing User:	Dr Mark Rubin
Date Deposited:	15 Dec 2023 03:03
Last Modified:	15 Dec 2023 03:03
Item ID:	22834
Official URL:	https://doi.org/10.31222/osf.io/3kv2b
DOI or Unique Handle:	10.31222/osf.io/3kv2b
Subjects:	General Issues > Data General Issues > Evidence General Issues > Explanation General Issues > Models and Idealization Specific Sciences > Probability/Statistics Specific Sciences > Psychology
Date:	10 December 2023
URI:	https://philsci-archive.pitt.edu/id/eprint/22834

Available Versions of this Item

• Type I error rates are not usually inflated. (deposited 15 Dec 2023 03:03) [Currently Displayed]
  • Type I Error Rates are Not Usually Inflated. (deposited 05 Jan 2024 03:06)

Monthly Views for the past 3 years

Monthly Downloads for the past 3 years

Plum Analytics

Altmetric.com

Actions (login required)

View Item