The true meaning/difference of alpha values and p-values

Question

EDIT: I may have been confused by the confusion of others. In any case, it helped a lot when I came to know that the $p$-value is stochastic. It does make sense, given the $p$-value is a transformation of the test statistic, which is stochastic!

So, what I gather so far (thanks for my statistics course for not being helpful!):

• The $\alpha$-value is the maximum chance of making a type I error given we assume the null hypothesis being correct AND the null is correct (I tried to simulate 10,000 $p$-values from two-sample t-tests for two ${N}(2,1)$ distributions (n = 3 or n =30).
• In this case, $p$ is also following a uniform distribution with $p$ between 0 and 1.
• If I calculate $p$-values when the null hypothesis cannot be true (like t-test of $N(2,1)$ vs $N(4,1)$; n = 3 or n = 30), the lower $p$-values tend to have a higher probability than the rest.
• If a person does an experiment with, say two samples of n = 30 and he tries to test the difference (or, rather, equality) of the means, it is perfectly possible to obtain a $p$-value > 0.05 purely by chance, even if the samples are truly different. Following the existing protocol in science, he accepts the null hypothesis (but it is debated why we use such an arbitrary limit!).
• Another scientist might try to replicate the experiment (or, more realistically, the first person will try to replicate the experiment because he can't publish this), and he gets, by pure chance, $p$ < 0.05 (say 0.03). Now he's in business.
• If you really want to see if the distribution of $p$ is inconsistent with the null hypothesis, you have to repeat the exact same experiment and analysis a lot of times!

So, the $\alpha$ value indicates how willing we are to make a type I error, assuming the null hypothesis is correct. It might be useful if we want to accept or reject, say, batches of a product. The $p$-value is something we calculate after we have done our experiments, and one single value does not seem to tell us much about how far we are from the null being true alternative being indistinguishable from the null (http://amstat.tandfonline.com/doi/pdf/10.1198/000313008X332421). Then we need to look at the whole distribution of $p$. Can we bootstrap us out of this?

Oh yeah, and regarding the CI. People tend to advise the CI is used instead of the p-value (unless they overlap and you have to check if they really are different). Do you agree?

------ RANDOM NONSENSE GIVEN BELOW --------

I have been puzzled by the seemingly great number of texts, that disagree on the usage of alpha-values and p-values. Even my own textbook looks like it mixes the philosophy and methods of two schools!

So I just want to really clarify what is going on - I'm not too strong in the statistics or math jargon, so I get lost in most articles beyond the simplest examples (like tossing a coin)! Therefore, I hope that you out there can help me out.

My current understanding goes like this: The alpha value is useful for the creation of confidence intervals (CIs) for the sample means, and an alpha value of for instance 0.05 (95% CI) will assure you, that only 5% of all future samples of the population will fail to contain the true mean in their CIs. This might be useful if you, for instance, continuously take samples from a production to check the product quality (discard the 5% of the batches when using 95% CI or change some production variables if >5% of samples disagree? I'm not sure.).

The big trouble is, that the alpha value is very often used mistakenly together with hypothesis testing, e.g. testing the difference of means between samples and using p < 0.05 as criteria. The p-value is interpreted as "there is a <5% chance that the effect is not significant, while it actually is (type I error?)". This, I have learned is wrong. The p-value rather describes that, given the null hypothesis is true, what the probability is to see a given or more extreme value. Like testing that the difference in means are zero, but when comparing to samples, the difference of the means might be e.g. 3 units by pure chance. I have stumbled upon some papers where they 'calibrate' p-values and relate them directly to the error rate, which is what many people actually mean by saying "p < 0.05". Here, you have to state the % of true nulls - I cannot readily interpret this, can anybody help? It seems it relates to, for instance, the empirical knowledge of how often an effect is seen as effective. Am I wrong?

Anyhow, for the error rate, I previously had a good idea about how to calculate it by Monte Carlo methods. But now I can't recall my reasoning! But I remember I figured the error rate depends on 1) sample sizes, 2) the distributions of the populations you sample, 3) how "far" the H0 and H1 hypotheses are from each other. You are less likely to be wrong if H0 states equal means, but the true means of the populations are far apart.

Cheers, Steffen

Answer 1

The big trouble is, that the alpha value is very often used mistakenly together with hypothesis testing,

Sorry, you don't have it correct there.

Indeed, $\alpha$ is fundamental to hypothesis testing. Explicitly, it is your chosen (maximum) type I error rate under the null hypothesis, and the basis on which the rejection region is chosen.

Let me start with a basic/general discussion of hypothesis testing with a more-or-less Neyman-Pearson flavor (but which is not formally NP).

Let's take as given that you want to test some hypothesis about some population characteristic, and you have a null hypothesis and an alternative hypothesis. Let's assume for now that the null is a point null but that the alternative is not.

1. You choose* some test statistic that will tend to behave differently when the alternative is true than when the null is true.

* there's theory that can help pick good ones if you know a lot about the population distribution but that's entirely beside the point here, and we rarely actually know that kind of information in any case.

2. You then compute the distribution of the test statistic when the null is true (perhaps by making assumptions about the population distribution and computing the sampling distribution of the test statistic when the null is true, or perhaps by making exchangeability assumptions and invoking some form of resampling for that purpose - such as randomization tests or bootstrap tests)

3. You identify a proportion $\alpha$ (or no more than $\alpha$) of the distribution that's more consistent with the alternative** than the null and call that your rejection region.

4. If your test statistic falls into the rejection region you reject the null hypothesis. If the null was actually true this happens rarely (i.e. with probability $\alpha$, whereas if the alternative is true it should happen considerably more often).

** However, it's possible to base a test purely on likelihood and take a more Fisherian-style point of view, where all the lowest-likelihood samples (under the null) compromise the rejection region.

Note that we haven't mentioned p-values at all yet, though they're common in the FIsherian-style approach. [However, they also fit in with the NP approach if you recognize that $p\leq\alpha$ precisely when the test statistic is in the rejection region.]

My current understanding goes like this: The alpha value is useful for the creation of confidence intervals (CIs)

No. Well, $\alpha$ does come into it in the sense that you choose a coverage probability of $(1-\alpha)$.

e.g. testing the difference of means between samples

Weren't you just calculating a CI? How did we jump to doing a test? What are you trying to do, produce a CI or test something?

The p-value is interpreted as "there is a <5% chance that the effect is not significant, while it actually is (type I error?)".

Can you show us someone saying exactly that? I can't quite follow what you're saying there.

This, I have learned is wrong.

Well, yes it's wrong, whatever it was trying to say.

I have stumbled upon some papers where they 'calibrate' p-values and relate them directly to the error rate,

I have no clear idea what you're saying there, but

a. in general don't try to learn statistics from what people do in papers, at least not until you have a solid grounding in it.

b. it's impossible to discuss a second hand report of what people do. If you want to discuss what you see in a paper, quote it and give a proper reference (so we can see more context if needed).

You have many confusions here. It might have been more useful to have given explicit examples of what you've found that were directly contradictory than present your own understanding.

Here, you have to state the % of true nulls

With point nulls, this is usually 0.

I cannot readily interpret this, can anybody help? It seems it relates to, for instance, the empirical knowledge of how often an effect is seen as effective. Am I wrong?

Yes, you're wrong. The word "effective" doesn't belong there. If your null is "no effect" it relates to how often the effect is completely absent.

But you have to be careful about what is being done here -- it sounds like someone is maybe taking a Bayesian approach but it's impossible to tell -- because again all we have is a somewhat muddled second hand report. It's impossible to untangle your misunderstandings from the misunderstandings of the people you're talking about.

Anyhow, for the error rate, I previously had a good idea about how to calculate it by Monte Carlo methods. But now I can't recall my reasoning!

It's not clear to me what you seek here.

While the first half of your question was answerable enough (by explaining some of your misconceptions), if you can clarify and narrow (and add context to) the later part of your question you might like to post that as one or two new questions.