An Explorable Multiverse Analysis of Durante et al. (2013)

Abstract

In this paper, we reproduce a small part of Steegen et al.'s multiverse analysis of Durante et al.'s study using explorable explanations. The data processing options can be selected interactively, which allows us to show the interaction plot reported in Durante et al. in addition to the p-value.

Author Keywords

Multiverse analysis.

ACM Classification Keywords

H5.2 User Interfaces: Evaluation/Methodology

General Terms

Human Factors; Design; Experimentation; Measurement.

Introduction

Steegen and colleagues introduced the concept of multiverse analysis, which they illustrated by re-analyzing data from a 2013 paper by Durante and colleagues entitled "The fluctuating female vote: Politics, religion, and the ovulatory cycle". Here, we report the initial part of the same multiverse analysis using an explorable explanation instead of p-value summaries. This article is meant to illustrate the use of explorable multiverse analysis at the data processing level.

Analysis

The default analysis below reflects the choices made by Durante et al. . Other options reflect alternatives considered by Steegen et al. . Much of the text below is copied from Steegen et al. , in order to give an idea of what their article could have looked liked had they used explorable explanations.

Fertility

The classification of women into a high or low fertility group based on cycle day can be done in several ways:

• Participants with cycle days ranging from 7 to 14 are assigned to the high fertility group, whereas participants with cycle days ranging from 17 to 25 are assigned to the low fertility group ,
• days 6–14 are used for high fertility, whereas days 17–27 are used for low fertility ,
• days 9–17 for high fertility and 18–25 for low fertility ,
• days 8–14 for high fertility and 1–7 and 15–28 for low fertility , and
• days 9–17 for high fertility and 1–8 and 18–28 for low fertility .

Second, there are different reasonable ways of estimating a woman's next menstrual onset, which is an intermediate step in determining cycle day. A woman's cycle day can be based on the number of days before next menstrual onset, which in turn is based on cycle length, which is computed as the difference between the start date of the woman’s last menstrual period and the start date of the woman's previous menstrual period . Another way to estimate next menstrual onset is based on the women's reported estimate of their typical cycle length .

Relationship status

There are at least three options for the dichotomization of women's relationship status into single or committed. Women who selected response Option 1 or 2 on the relationship status item can be assigned to the group of single women, whereas women who selected response Option 3 or 4 can be assigned to the group of women in committed relationships. Due to the ambiguous nature of response Option 2 (dating or involved with only one partner), women who select this response could reasonably be classified as being in committed relationships. A third option involves discarding participants who select this ambiguous response option, and only classifying participants selecting Option 1 as single women, and participants selecting Option 3 or 4 as women in relationships.

Exclusion criteria

The assignment of the participants to a high or low fertility group automatically excludes women whose cycle days are not in the high or low fertility range. One option is not to exclude any participant beyond this exclusion . Alternatively, it is not unreasonable to exclude participants with irregular cycle lengths. This could amount to only including women with cycle lengths 25 to 35 . This exclusion criterion can be instantiated using a woman's computed cycle length. Another reasonable way of instantiating this exclusion criterion is by using a woman's self-reported typical cycle length.

Another justifiable exclusion criterion concerns women's reported certainty ratings of the start dates of their last two menstrual periods. One option is not to exclude these participants . It is also reasonable to exclude participants who were not sufficiently confident about their report and to consider only data from participants with a certainty rating above the midpoint for both dates .

Results

The interaction between relationship status and fertility in study 1 is shown in Figure 1. This plot reproduces Figure 1 from Durante et al.'s article but with the y-axis starting at zero. This figure does not appear Steegen et al.'s multiverse analysis , as there would be 180 such figures to show, which would be impractical with a static paper. The p-value for the interaction is also shown on Figure 1.

Discussion

As in the original paper, we can see that "the multiverse analysis revealed that almost all choice combinations for data processing lead to large p values" and we can again conclude that "the effect of fertility on religion seems too sensitive to arbitrary choices and thus too fragile to be taken seriously" . Figure 1 can animated by holding the 'A' key, giving a striking demonstration of the variability of effect sizes across the multiverse that can usefully complement Steegen et al's histogram of p-values.