EDIT: I may have been confused by the confusion of others. In any case, it helped a lot when I came to know that the $p$-value is stochastic. It does make sense, given the $p$-value is a transformation of the test statistic, which is stochastic!
So, what I gather so far (thanks for my statistics course for not being helpful!):
- The $\alpha$-value is the maximum chance of making a type I error given we assume the null hypothesis being correct AND the null is correct (I tried to simulate 10,000 $p$-values from two-sample t-tests for two ${N}(2,1)$ distributions (n = 3 or n =30).
- In this case, $p$ is also following a uniform distribution with $p$ between 0 and 1.
- If I calculate $p$-values when the null hypothesis cannot be true (like t-test of $N(2,1)$ vs $N(4,1)$; n = 3 or n = 30), the lower $p$-values tend to have a higher probability than the rest.
- If a person does an experiment with, say two samples of n = 30 and he tries to test the difference (or, rather, equality) of the means, it is perfectly possible to obtain a $p$-value > 0.05 purely by chance, even if the samples are truly different. Following the existing protocol in science, he accepts the null hypothesis (but it is debated why we use such an arbitrary limit!).
- Another scientist might try to replicate the experiment (or, more realistically, the first person will try to replicate the experiment because he can't publish this), and he gets, by pure chance, $p$ < 0.05 (say 0.03). Now he's in business.
- If you really want to see if the distribution of $p$ is inconsistent with the null hypothesis, you have to repeat the exact same experiment and analysis a lot of times!
So, the $\alpha$ value indicates how willing we are to make a type I error, assuming the null hypothesis is correct. It might be useful if we want to accept or reject, say, batches of a product.
The $p$-value is something we calculate after we have done our experiments, and one single value does not seem to tell us much about how far we are from the null being true alternative being indistinguishable from the null (http://amstat.tandfonline.com/doi/pdf/10.1198/000313008X332421). Then we need to look at the whole distribution of $p$. Can we bootstrap us out of this?
Oh yeah, and regarding the CI. People tend to advise the CI is used instead of the p-value (unless they overlap and you have to check if they really are different). Do you agree?
------ RANDOM NONSENSE GIVEN BELOW --------
I have been puzzled by the seemingly great number of texts, that disagree on the usage of alpha-values and p-values. Even my own textbook looks like it mixes the philosophy and methods of two schools!
So I just want to really clarify what is going on - I'm not too strong in the statistics or math jargon, so I get lost in most articles beyond the simplest examples (like tossing a coin)! Therefore, I hope that you out there can help me out.
My current understanding goes like this: The alpha value is useful for the creation of confidence intervals (CIs) for the sample means, and an alpha value of for instance 0.05 (95% CI) will assure you, that only 5% of all future samples of the population will fail to contain the true mean in their CIs. This might be useful if you, for instance, continuously take samples from a production to check the product quality (discard the 5% of the batches when using 95% CI or change some production variables if >5% of samples disagree? I'm not sure.).
The big trouble is, that the alpha value is very often used mistakenly together with hypothesis testing, e.g. testing the difference of means between samples and using p < 0.05 as criteria. The p-value is interpreted as "there is a <5% chance that the effect is not significant, while it actually is (type I error?)". This, I have learned is wrong. The p-value rather describes that, given the null hypothesis is true, what the probability is to see a given or more extreme value. Like testing that the difference in means are zero, but when comparing to samples, the difference of the means might be e.g. 3 units by pure chance. I have stumbled upon some papers where they 'calibrate' p-values and relate them directly to the error rate, which is what many people actually mean by saying "p < 0.05". Here, you have to state the % of true nulls - I cannot readily interpret this, can anybody help? It seems it relates to, for instance, the empirical knowledge of how often an effect is seen as effective. Am I wrong?
Anyhow, for the error rate, I previously had a good idea about how to calculate it by Monte Carlo methods. But now I can't recall my reasoning! But I remember I figured the error rate depends on 1) sample sizes, 2) the distributions of the populations you sample, 3) how "far" the H0 and H1 hypotheses are from each other. You are less likely to be wrong if H0 states equal means, but the true means of the populations are far apart.
Cheers, Steffen