Rubin, Mark (2026) p-hacking inflates Type I error rates in the error statistical approach but not in the formal inference approach. [Preprint]

Text 2602.21792v2.pdf - Other Available under License Creative Commons Attribution. Download (562kB)

Abstract

p-hacking occurs when researchers conduct multiple significance tests (e.g., p1;H0,1 and p2;H0,2) and then selectively report tests that yield desirable (usually significant) results (e.g., p2 ≤ 0.05;H0,2) without correcting for multiple testing (e.g., 0.05/2 = 0.025). In the present article, I consider p-hacking in the context of two philosophies of significance testing — the error statistical approach and the formal inference approach. I argue that although p-hacking inflates Type I error rates in the error statistical approach, it does not inflate them in the formal inference approach. Specifically, in the error statistical approach, the “actual” familywise error rate (e.g., 1 − [1 − 0.05]2 = 0.098 for two tests) is relevant because it covers both the selectively reported and unreported tests in the “actual” test procedure (i.e., p1;H0,1 and p2;H0,2). In this approach, Type I error rate inflation occurs because the “actual” error rate (0.098) is higher than the nominal error rate (0.05). In contrast, in the formal inference approach, the “actual” familywise error rate is irrelevant because (a) the researcher does not report a statistical inference about the corresponding intersection null hypothesis (i.e., H0,1 ∩ H0,2), and (b) the “actual” familywise error rate does not license inferences about the reported individual hypotheses (i.e., H0,2). Instead, in the formal inference approach, only the nominal error rate is relevant, and a comparison with the “actual” error rate is inappropriate. Implications for conceptualizing, demonstrating, and reducing p-hacking are discussed.

---

Export/Citation:

EndNote | BibTeX | Dublin Core | ASCII/Text Citation (Chicago) | HTML Citation | OpenURL

Social Networking:

Share |

---

Item Type:	Preprint
Creators:	Creators Email ORCID Rubin, Mark Mark-Rubin@outlook.com 0000-0002-6483-8561
Keywords:	error statistics; familywise error rate; null results; p-hacking; questionable research practices; Texas sharpshooter fallacy
Subjects:	General Issues > Evidence General Issues > Models and Idealization
Depositing User:	Dr Mark Rubin
Date Deposited:	17 Mar 2026 11:20
Last Modified:	17 Mar 2026 11:20
Item ID:	28637
Official URL:	https://doi.org/10.48550/arXiv.2602.21792
DOI or Unique Handle:	10.48550/arXiv.2602.21792
Subjects:	General Issues > Evidence General Issues > Models and Idealization
Date:	3 March 2026
URI:	https://philsci-archive.pitt.edu/id/eprint/28637

Monthly Views for the past 3 years

Monthly Downloads for the past 3 years

Plum Analytics

Altmetric.com

Actions (login required)

View Item