0
$\begingroup$

Apologies if the answer to this is obvious, but I can't understand why the usual interpretation of the p-value (probability under the null model of obtaining a test statistic at least as unlikely as the one observed) is valid in a simple one-sample z-test.

As an example, say have some data with mean 2 and std of the mean of 1, and we want to test if the data is greater than 0. This gives a z-score of 2, which we can look up in a table to obtain p=0.023. This corresponds to the shaded area below.

AUC

What confuses me is that this seems to be the probability of drawing at most the mean of the null model (0) given a distribution inferred from the data (N(2, 1)), whereas the usual meaning of the p-value would be the probability of drawing at least 2 from a null model as illustrated below:

AUC2

I can see that these are the same if we assume the null model to have the same width as the distribution we obtain from the data. Is this just an assumption of the test, or is there some deep reason to expect this to be the case?

$\endgroup$
0
$\begingroup$

For $a>0$, \begin{align*} \mathbb{P}(N(a,1)<0) &= \mathbb{P}(N(0,1) + a <0) \\ &= \mathbb{P}(N(0,1)<-a) \\ &= \mathbb{P}(N(0,1)>a) \\ \end{align*}

The test statistic $$ Z:= \sqrt{n} \frac{\bar{X}_n}{\hat{S}} $$ has distribution $N(a\sqrt{n},1)$.

However, for the test procedure it does not mean that the type I and type II errors are the same (i.e that $P_{H_0}(H_1) = P_{H_1}(H_0)$) in that the null will be rejected only if $\mid Z \mid$ is larger than $q_{1-\alpha/2}$ (for a fixed type I error $\alpha$) which might not be even close to $a\sqrt{n}$

For example if you take $n=1$ and $X \sim N(2,1)$ and $\alpha = 0.05$ ($q_{1-\alpha/2}\approx 1.96$) then $\beta \approx 51\%$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm sorry but I don't understand how this answers the question? I follow your first calculations but if I understand correctly, they just restate that under the assumption that the null has the same width as the observed data, the two areas are the same? $\endgroup$ – ahura Feb 15 '19 at 19:04
  • $\begingroup$ I might have misunderstood your question. Indeed this property doesn't stand anymore if distributions have different variances. I don't understand what you mean with " Is this just an assumption of the test" ? $\endgroup$ – winperikle Feb 15 '19 at 20:01
  • $\begingroup$ I mean is it an assumption of the Z-test that the variance is the same. $\endgroup$ – ahura Feb 15 '19 at 20:43
  • $\begingroup$ I might be wrong but I don't think this is a crucial assumption. Take for example the exponential distribution and remove the standard deviation from Z. Then under alternatives hypotheses Z will have a different variance. However when performing your test you will need to compare the value of Z with the quantile of a normal distribution of mean 0 but with a variance equals to the one estimated from the sample. If you standardized Z as usualy done you will always compare it with quantiles from the standard normal distriubtion. This is a result of the central limit theorem. $\endgroup$ – winperikle Feb 15 '19 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.