I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.
Here is the definition of p-value from Wikipedia:
The p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. (http://en.wikipedia.org/wiki/P-value)
My first question pertains to the expression "at least as extreme as the one that was actually observed." My understanding of the logic underlying the use of p-value is the following: If the p-value is small, it's unlikely that the observation occurred assuming the null hypothesis and we may need an alternative hypothesis to explain the observation. If the p-value is not so small, it is likely that the observation occurred only assuming the null hypothesis and the alternative hypothesis is not necessary to explain the observation. So if someone wants to insist on a hypothesis he/she has to show that the p-value of the null hypothesis is very small. With this view in mind, my understanding of the ambiguous expression is that p-value is $\min[P(X<x),P(x<X)]$, if the PDF of the statistic is unimodal, where $X$ is the test statistic and $x$ is its value obtained from the observation. Is this right? If it is right, is it still applicable to use the bimodal PDF of the statistic? If two peaks of the PDF are separated well and the observed value is somewhere in the low probability density region between the two peaks, which interval does the p-value give the probability of?
The second question is about another definition of p-value from Wolfram MathWorld:
The probability that a variate would assume a value greater than or equal to the observed value strictly by chance. (http://mathworld.wolfram.com/P-Value.html)
I understood that the phrase "strictly by chance" should be interpreted as "assuming a null hypothesis". Is that right?
The third question regards the use of "null hypothesis". Let's assume that someone wants to insist that a coin is fair. He expresses the hypothesis as that relative frequency of heads is 0.5. Then the null hypothesis is "relative frequency of heads is not 0.5." In this case, whereas calculating the p-value of the null hypothesis is difficult, the calculation is easy for the alternative hypothesis. Of course the problem can be resolved by interchanging the role of the two hypotheses. My question is that rejection or acceptance based directly on the p-value of the original alternative hypothesis (without introducing the null hypothesis) is whether it is OK or not. If it is not OK, what is usual workaround for such difficulties when calculating the p-value of a null hypothesis?
I posted a new question that is more clarified based on the discussion in this thread.