Rubin, Mark (2021) When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese. ISSN 0039-7857

Preview

Text Rubin (2021) When to Adjust Alpha.pdf - Updated Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (944kB) | Preview

Abstract

Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.

---

Export/Citation:

EndNote | BibTeX | Dublin Core | ASCII/Text Citation (Chicago) | HTML Citation | OpenURL

Social Networking:

Share |

---

Item Type:	Published Article or Volume
Creators:	Creators Email ORCID Rubin, Mark Mark-Rubin@outlook.com 0000-0002-6483-8561
Keywords:	experimentwise error; familywise error; multiple testing; multiple comparisons; simultaneous testing; Type I error
Subjects:	General Issues > Evidence Specific Sciences > Probability/Statistics
Depositing User:	Dr Mark Rubin
Date Deposited:	08 Jul 2021 00:37
Last Modified:	08 Jul 2021 00:37
Item ID:	19294
Journal or Publication Title:	Synthese
Official URL:	https://doi.org/10.1007/s11229-021-03276-4
DOI or Unique Handle:	10.1007/s11229-021-03276-4
Subjects:	General Issues > Evidence Specific Sciences > Probability/Statistics
Date:	7 July 2021
ISSN:	0039-7857
URI:	https://philsci-archive.pitt.edu/id/eprint/19294

Monthly Views for the past 3 years

Monthly Downloads for the past 3 years

Plum Analytics

Altmetric.com

Actions (login required)

View Item