2
$\begingroup$

I am reading some books about hypothesis testing, but I am not sure if my following reasoning makes sense:

Assume I have a gaussian random variable $X \sim N(\mu, \sigma)$ with $\sigma=1$. Now I obtain 5 iid samples, $x_1, \cdots, x_5$.

I want to check if $\mu<0$. So I set up the null hypothesis and alternative hypothesis to be $H_0: \mu=0$ and $H_1: \mu<0$

Therefore, for each sample $x_i$, I can compute the p-value = $P(X_i\le x_i)$, and denote it by $\alpha$. Therefore, I have $(1-\alpha)$ confidence to reject $H_0$. Also, by using the rule $(X\le x_i)$ to reject $H_0$, I have type-I error equal to $\alpha$.

Now based on 5 samples, I have 5 p-values $\alpha_i$, ($i=1, \cdots, 5$). Therefore, I have $P(X_1\le x_1, \cdots, X_5\le x_5) = \alpha_1 \times \alpha_2 \times \cdots \times \alpha_5$. Therefore, using the decision rule that $(X_1\le x_1, \cdots, X_5\le x_5)$ to reject H0, I have a type-I error equal to $\times \alpha_1 \times \cdots \times \alpha_5$. And therefore, I have $(1-\times \alpha_1 \times \cdots \times \alpha_5)$ confidence to reject $H_0$.

Basically, I want to use the 5 $x_i$'s for future testing. Next time I obtain 5 samples, I'll compare them to $x_i$. And I want to see how much confidence level this decision rule gives me. It seems to me that textbooks usually compute the p-value for one sample xi. I am basically trying to compute the "p-value" for 5 samples.

Is the above reasoning correct?

Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ Hmmm...How would you interpret the negative confidence that arises if, perchance, $5! \alpha_1 \times \cdots \times \alpha_5 \gt 1$? (This will happen if, for instance, all five $x$'s exceed $-0.3\sigma$, which occurs almost $9$% of the time by chance.) $\endgroup$
    – whuber ♦
    Commented Sep 9, 2012 at 22:23
  • $\begingroup$ @whuber, that's a good point. I don't know how to interpret that, but maybe I can change my decision rule to if $(X_1\le x_1, \cdots, X_5\le x_5)$, then reject $H_0$. In that case, I have a type-I error of $\alpha_1 \times \cdots \times \alpha_5$ which is always smaller than 1, and never leads to negative confidence. $\endgroup$
    – calbear
    Commented Sep 9, 2012 at 23:02
  • $\begingroup$ I'm confused about what you're doing. Didn't you indicate that the $x_i$ are your sample results? Doesn't that constitute your entire dataset? Then what does $X_i\le x_i$ mean? What do the $X_i$ represent? $\endgroup$
    – whuber ♦
    Commented Sep 10, 2012 at 15:43
  • $\begingroup$ sorry for not explaining well. $x_i$ is the i-th observed sample value, $X_i$ is the random variable of the i-th sample. $X_i$'s are iid, and satisfies $N(\mu, \sigma)$. $\endgroup$
    – calbear
    Commented Sep 10, 2012 at 16:17
  • 1
    $\begingroup$ OK, you definitely need to edit your question to incorporate this last clarification: otherwise you will likely get answers that don't address your situation and so they might confuse you. The name for what you are doing, by the way, is "prediction limit." $\endgroup$
    – whuber ♦
    Commented Sep 10, 2012 at 16:41

1 Answer 1

Reset to default
1
$\begingroup$

Hypothesis testing does not use individual observations for testing. That would involve too much variability to expect a good confidence interval (i.e. sufficiently narrow) or provide any reasonable power to reject the null hpyothesis under any reasonable alternative. Instead summary statistics that are functions of several samples are used. In your case when testing the mean = 0 vs mean < 0 for a sample of five normally distributed observations you would compute the sample mean divided by its standard deviation (in your case 1/√5) and compare it to values in the lower tail of a N(0,1) distribution. You then get a single p-value rather than combining 5 tests based on individual observations as you suggest for your test.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Michael. As a follow-up question, maybe what I am trying to do is not called "hypothesis testing", but I can certainly design decision rules that give me a very small type-I error. The question is why does it involve more variability than summary statistics? In some way, my decision rule takes into consideration all sample values. (Is there anything in statistics that studies whatever I am trying to do?) $\endgroup$
    – calbear
    Commented Sep 11, 2012 at 15:03
  • $\begingroup$ @calbear You start by doing five separate hypothesis tests. Each one is much more variable than a test based on the mean of the five samples (by the factor of √5 that I mentioned). So if the mean is actually <0 the p-values for the individual test should all be similar and a lot higher than for the mean of the five. Your so-called decision rule allows five distinct cutoff points. But if the samples are iid there would be no reason not to use the same cutoff point rahter than using the observed value as the cutoff. $\endgroup$
    – Michael R. Chernick
    Commented Sep 11, 2012 at 15:13
  • $\begingroup$ I do not think your rule is very good and I doubt that there is any statistical literature on it. There are as I said better and actually and optimal (in the sense of most powerful) test that I described based on the sample mean. $\endgroup$
    – Michael R. Chernick
    Commented Sep 11, 2012 at 15:14
  • $\begingroup$ Thanks for the comments. I mostly agree with you, but I am still not quite convinced... $\endgroup$
    – calbear
    Commented Sep 11, 2012 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.