8
$\begingroup$

A p-value is the probability to obtain a statistic that is at least as extreme as the one observed in the sample data when assuming that the null-hypothesis ($H_0$) is true.

Graphically this corresponds to the area defined by the sample statistic under the sampling distribution which one would obtain when assuming $H_0$:

center h0

However, because the shape of this assumed distribution is actually based on the sample data, centering it on $\mu_0$ seems like an odd choice to me.
If one would instead use the sampling distribution of the statistic, i.e. center the distribution on the sample statistic, then hypothesis testing would correspond to estimating the probability of $\mu_0$ given the samples.

center h1

In that case the p-value is the probability to obtain a statistic at least as extreme as $\mu_0$ given the data instead of the above definition.

Additionally, such an interpretation has the advantage of relating well to the concept of confidence intervals:
A hypothesis test with significance level $\alpha$ would be equivalent to checking whether $\mu_0$ falls within the $(1-\alpha)$ confidence interval of the sampling distribution.

CI2 95

I thus feel that centering the distribution on $\mu_0$ could be an unneccessary complication.
Are there any important justifications for this step which I did not consider?

$\endgroup$
  • 2
    $\begingroup$ Please tell us what the sampling distribution will be if you do not assume $H_0$. (Answer: you cannot, except in textbook examples where the alternative hypothesis specifies a unique distribution.) $\endgroup$ – whuber Jun 24 '16 at 14:15
  • $\begingroup$ I am not sure if I understand the request correctly but in the above example it would be the sampling distribution of the mean. I have now added a figure to the question which shows this distribution together with a 95% confidence interval/area which should also help illustrating the relationship to confidence intervals. $\endgroup$ – matti Jun 24 '16 at 15:22
  • 2
    $\begingroup$ You have no way to know the sampling distribution of the mean. To know that, you need to know the true mean: but that's precisely the quantity you are trying to test! Your logic is completely circular. $\endgroup$ – whuber Jun 24 '16 at 15:26
  • 1
    $\begingroup$ I understood that was your meaning. In general, until you know--or assume--the true parameters of the distribution, you cannot know the distribution of any property of the sample. (In fact, if you could deduce the distribution of any sample property without assuming knowledge of the parameters, that would be proof that it gives you no information about the parameters!) $\endgroup$ – whuber Jun 24 '16 at 15:52
  • 1
    $\begingroup$ I cannot, because it appears you are not using terms like "mean," "estimated," or even "H0" in their usual statistical senses. I am at a complete loss to understand even what your question is. The only thing that is clear is that it is predicated on a misunderstanding of null hypothesis testing, but your responses to my comments haven't provided any useful indications of what that misunderstanding might be. $\endgroup$ – whuber Jun 24 '16 at 16:26
5
$\begingroup$

Suppose $\boldsymbol X = (X_1, X_2, \ldots, X_n)$ is a sample drawn from a normal distribution with unknown mean $\mu$ and known variance $\sigma^2$. The sample mean $\bar X$ is therefore normal with mean $\mu$ and variance $\sigma^2/n$. On this much, I think there can be no possibility of disagreement.

Now, you propose that our test statistic is $$Z = \frac{\bar X - \mu}{\sigma/\sqrt{n}} \sim \operatorname{Normal}(0,1).$$ Right? BUT THIS IS NOT A STATISTIC. Why? Because $\mu$ is an unknown parameter. A statistic is a function of the sample that does not depend on any unknown parameters. Therefore, an assumption must be made about $\mu$ in order for $Z$ to be a statistic. One such assumption is to write $$H_0 : \mu = \mu_0, \quad \text{vs.} \quad H_1 : \mu \ne \mu_0,$$ under which $$Z \mid H_0 = \frac{\bar X - \mu_0}{\sigma/\sqrt{n}} \sim \operatorname{Normal}(0,1),$$ which is a statistic.

By contrast, you propose to use $\mu = \bar X$ itself. In that case, $Z = 0$ identically, and it is not even a random variable, let alone normally distributed. There is nothing to test.

$\endgroup$
  • 1
    $\begingroup$ Thank you. This is very straightforward and now I really wonder how I could have missed that before. All that would be left now as an excuse for the second presented case is to rely on the confidence interval calculation. However, because there the margin of error is explicitly added/subtracted from the mean or point estimate, the use of that estimate becomes a step that would need to be justified. $\endgroup$ – matti Jun 24 '16 at 21:12
12
$\begingroup$

However, because the shape of this assumed distribution is actually based on the sample data, centering it on H0 seems like an odd choice to me.

This is actually not true. The shape of this assumed distribution comes from accepting $H_0$ as true. Sample is not directly involved in that, other than by some assumptions. Using the sample directly, is not enough. You need also the null hypothesis to hold.

If one would instead use the sampling distribution of the statistic, i.e. center the distribution on the sample statistic, then hypothesis testing would correspond to estimating the probability of H0 given the samples.

The question is: how do you estimate a probability of something which you assume is true. In our case if you assume $H_0$ as true, is futile to try to estimate the probability that $H_0$ is true.

I thus feel that centering the distribution on H0 is an unneccessary complication.

You don't have two distributions there, there is only one, the one assumed to be your ground truth, aka the one which comes with $H_0$. There is however a sampling distribution derived from sample, but this is not involved in the hypotheses you use.

I good exercise would be to try to replicate the same logic with an asymmetric distribution. Take chi-square distribution like in chi square independence test. Are you able to reproduce it? I think the answer is no.

$\endgroup$
  • $\begingroup$ "This is actually not true. The shape of this assumed distribution comes from accepting H0 as true. Sample is not directly involved in that, other than by some assumptions." But in the case of the one sample t-test presented above, the test statistic includes the SEM and the sample mean and is thus dependent on the sample data. $$t = \frac{\bar{x}-\mu_0}{\frac{s}{{\sqrt{n}}}}$$ Furthermore the degrees of freedom which determine the height of the tails depend on the sample size. $\endgroup$ – matti Jun 24 '16 at 11:57
  • 1
    $\begingroup$ My formulation was misleading. I was trying to say that you can use any information you have, also the sample itself, but it is not enough. To evaluate p-values and to have a distribution you need to assume also the null hypothesis. I reformulate in the post also. $\endgroup$ – rapaio Jun 24 '16 at 21:02
  • 1
    $\begingroup$ ... Take for example your formula for $t$, it uses $\mu_0$ which I suppose it the value from the null hypothesis $H_0 : \mu = \mu_0$ $\endgroup$ – rapaio Jun 24 '16 at 21:14
1
$\begingroup$

From what I gather, you are arguing that it makes more sense to 'flip' $H_0$ and $H_1$.

I find it helpful to think of hypothesis testing as a proof by contradiction. We assume $H_0$ to be true, then show that evidence indicates such an assumption is flawed, thus justifying the rejection of $H_0$ in favor of $H_1$.

This works because when we assume $H_0$ and center our distribution there, we can determine how likely/unlikely our observation is. For example, if $H_0: \mu = 0$ vs. $H_1: \mu \neq 0$ and we determine from our testing that there is a less that 5% chance that the true mean $\mu$ actually equals 0, we can reject $H_0$ with 95% confidence.

The reverse is not necessarily true. Say we do an experiment and determine that there is actually a 30% chance that the null hypothesis still holds. We cannot reject the null, but we also do not accept it. This situation does not show that $H_0$ (the null) is true, but that we do not have the evidence to show that it is false.

Now imagine if we flipped this situation. Say we assume $H_1$ and find that given our results, the likelihood of $H_0$ is 5% or less, what does that mean? Sure we can reject the null, be can we necessarily accept $H_1$? It is hard to justify accepting the thing we assumed to be true in the beginning.

Showing that $H_0$ is false is not the result we are after; we want to argue in favor of $H_1$. By doing the test in the way you describe, we are showing that we do not have evidence to say that $H_1$ is false, which is subtly different than arguing $H_1$ is true.

$\endgroup$
  • $\begingroup$ Since the hypothesis test doesn't allow us to completely eliminate uncertainty I would't see it as a proof. Perhaps I did not make my point clear enough but I am essentially asking for a logical rather than a semantic reason to shift the sampling distribution to $H_0$. $\endgroup$ – matti Jun 24 '16 at 15:39
  • $\begingroup$ And in general, H1 is pretty vague (mu != 0), making likelihood calculations problematic. Though I suppose that is often a good incentive for people to go Bayesian. :) $\endgroup$ – Hao Ye Jun 24 '16 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.