# If the distribution of test statistic is bimodal, does p-value mean anything?

P-value is defined the probability of obtaining a test-statistic at least as extreme as what is observed, assuming null-hypothesis is true. In other words,

$$P( X \ge t | H_0 )$$ But what if the test-statistic is bimodal in distribution? does p-value mean anything in this context? For example, I am going to simulate some bimodal data in R:

set.seed(0)
# Generate bi-modal distribution
bimodal <- c(rnorm(n=100,mean=25,sd=3),rnorm(n=100,mean=100,sd=5))
hist(bimodal, breaks=100)


And let's assume we observe a test statistic value of 60. And here we know from the picture this value is very unlikely. So ideally, I would want a statistic procedure that I use(say, p-value) to reveal this. But if we compute for the p-value as defined, we get a pretty high-p value

observed <- 60

# Get P-value
sum(bimodal[bimodal >= 60])/sum(bimodal)
[1] 0.7991993


If I did not know the distribution, I would conclude that what I observed is simply by random chance. But we know this is not true.

I guess the question I have is: Why, when computing p-value, do we compute the probability for the values "at least as extreme as" the observed? And if I encounter a situation like the one I simulated above, what is the alternative solution?

• Welcome to the wonderful world of Null Hypothesis Significance Testing! Seriously: I honestly can't think of a test statistic that has a bimodal distribution under the null hypothesis (which is the one that we care about in NHST). So +1 for an interesting question, but I kind of doubt its practical relevance... unless you have a specific example in mind? – S. Kolassa - Reinstate Monica Mar 19 '14 at 15:42
• I agree with @StephanKolassa ; there are certainly distributions of data that are bimodal, but what sort of test statistic is? – Peter Flom - Reinstate Monica Mar 19 '14 at 15:47
• I would disagree with the characterization of p-values suggested by the first formula. The correct sense of "at least as extreme" in the Neyman-Pearson theory is in terms of relative likelihood and not in terms of the usual ordering of the reals (as indicated in the formula). The two are equivalent in many standard testing situations but differ sharply when the sampling distribution is bimodal. This distinction will therefore resolve the question satisfactorily, I think. – whuber Mar 19 '14 at 16:38
• @whuber Can you please elaborate on this a little bit, maybe with a simple example? – Szabolcs Jun 22 '14 at 13:24
• @Szabolcs Let $G_\theta$ be a Beta$(\theta,\theta)$ distribution and for $\theta\ge 1$ let $F_\theta(x)$ be an equal mixture of $G_\theta(x)$ and $G_\theta(-x)$ ($x \in [-1,1]$). The PDF of $F_1$ is uniform while the PDF of, say, $F_2$ is bimodal with peaks at $\pm 1/2$. Suppose $X\sim F_\theta$. The rejection region for the LR test of $H_0: X\sim F_1$ vs $H_A: X\sim F_2$ consists of two intervals far from the extremes $\pm 1$--one around $1/2$ and the other around $-1/2$--because evidence for $\theta=2$ is strongest there. – whuber Jun 22 '14 at 14:20