12
$\begingroup$

P-value is defined the probability of obtaining a test-statistic at least as extreme as what is observed, assuming null-hypothesis is true. In other words,

$$P( X \ge t | H_0 )$$ But what if the test-statistic is bimodal in distribution? does p-value mean anything in this context? For example, I am going to simulate some bimodal data in R:

set.seed(0)
# Generate bi-modal distribution
bimodal <- c(rnorm(n=100,mean=25,sd=3),rnorm(n=100,mean=100,sd=5)) 
hist(bimodal, breaks=100)

enter image description here

And let's assume we observe a test statistic value of 60. And here we know from the picture this value is very unlikely. So ideally, I would want a statistic procedure that I use(say, p-value) to reveal this. But if we compute for the p-value as defined, we get a pretty high-p value

observed <- 60

# Get P-value
sum(bimodal[bimodal >= 60])/sum(bimodal)
[1] 0.7991993

If I did not know the distribution, I would conclude that what I observed is simply by random chance. But we know this is not true.

I guess the question I have is: Why, when computing p-value, do we compute the probability for the values "at least as extreme as" the observed? And if I encounter a situation like the one I simulated above, what is the alternative solution?

|cite|improve this question
$\endgroup$
  • 7
    $\begingroup$ Welcome to the wonderful world of Null Hypothesis Significance Testing! Seriously: I honestly can't think of a test statistic that has a bimodal distribution under the null hypothesis (which is the one that we care about in NHST). So +1 for an interesting question, but I kind of doubt its practical relevance... unless you have a specific example in mind? $\endgroup$ – S. Kolassa - Reinstate Monica Mar 19 '14 at 15:42
  • 1
    $\begingroup$ I agree with @StephanKolassa ; there are certainly distributions of data that are bimodal, but what sort of test statistic is? $\endgroup$ – Peter Flom - Reinstate Monica♦ Mar 19 '14 at 15:47
  • 7
    $\begingroup$ I would disagree with the characterization of p-values suggested by the first formula. The correct sense of "at least as extreme" in the Neyman-Pearson theory is in terms of relative likelihood and not in terms of the usual ordering of the reals (as indicated in the formula). The two are equivalent in many standard testing situations but differ sharply when the sampling distribution is bimodal. This distinction will therefore resolve the question satisfactorily, I think. $\endgroup$ – whuber♦ Mar 19 '14 at 16:38
  • $\begingroup$ @whuber Can you please elaborate on this a little bit, maybe with a simple example? $\endgroup$ – Szabolcs Jun 22 '14 at 13:24
  • 2
    $\begingroup$ @Szabolcs Let $G_\theta$ be a Beta$(\theta,\theta)$ distribution and for $\theta\ge 1$ let $F_\theta(x)$ be an equal mixture of $G_\theta(x)$ and $G_\theta(-x)$ ($x \in [-1,1]$). The PDF of $F_1$ is uniform while the PDF of, say, $F_2$ is bimodal with peaks at $\pm 1/2$. Suppose $X\sim F_\theta$. The rejection region for the LR test of $H_0: X\sim F_1$ vs $H_A: X\sim F_2$ consists of two intervals far from the extremes $\pm 1$--one around $1/2$ and the other around $-1/2$--because evidence for $\theta=2$ is strongest there. $\endgroup$ – whuber♦ Jun 22 '14 at 14:20
5
$\begingroup$

What makes a test statistic "extreme" depends on your alternative, which imposes an ordering (or at least a partial order) on the sample space - you seek to reject those cases most consistent (in the sense being measured by a test statistic) with the alternative.

When you don't really have an alternative to give you a something to be most consistent with, you're essentially left with the likelihood to give the ordering, most often seen in Fisher's exact test. There, the probability of the outcomes (the 2x2 tables) under the null orders the test statistic (so that 'extreme' is 'low probability').

If you were in a situation where the far left (or far right, or both) of your bimodal null distribution was associated with the kind of alternative you were interested in, you wouldn't seek to reject a test statistic of 60. But if you're in a situation where you don't have an alternative like that, then 60 is unsual - it has low likelihood; a value of 60 is inconsistent with your model and would lead you to reject.

[This would be seen by some as one central difference between Fisherian and Neyman-Pearson hypothesis testing. By introducing an explicit alternative, and a ratio of likelihoods, a low likelihood under the null won't necessarily cause you to reject in a Neyman-Pearson framework (as long as it performs relatively well compared too the alternative), while for Fisher, you don't really have an alternative, and the likelihood under the null is the thing you're interested in.]

I'm not suggesting either approach is right or wrong here - you go ahead and work out for yourself what kind of alternatives you seek power against, whether it's a specific one, or just anything that's unlikely enough under the null. Once you know what you want, the rest (including what 'at least as extreme' means) pretty much follows from that.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.