P-value is defined the probability of obtaining a test-statistic at least as extreme as what is observed, assuming null-hypothesis is true. In other words,
$$P( X \ge t | H_0 )$$ But what if the test-statistic is bimodal in distribution? does p-value mean anything in this context? For example, I am going to simulate some bimodal data in R:
set.seed(0)
# Generate bi-modal distribution
bimodal <- c(rnorm(n=100,mean=25,sd=3),rnorm(n=100,mean=100,sd=5))
hist(bimodal, breaks=100)
And let's assume we observe a test statistic value of 60. And here we know from the picture this value is very unlikely. So ideally, I would want a statistic procedure that I use(say, p-value) to reveal this. But if we compute for the p-value as defined, we get a pretty high-p value
observed <- 60
# Get P-value
sum(bimodal[bimodal >= 60])/sum(bimodal)
[1] 0.7991993
If I did not know the distribution, I would conclude that what I observed is simply by random chance. But we know this is not true.
I guess the question I have is: Why, when computing p-value, do we compute the probability for the values "at least as extreme as" the observed? And if I encounter a situation like the one I simulated above, what is the alternative solution?