We've got into an argument with colleague, about something that seems to basic stats question.

Suppose that you measure some metric for two samples, A and B. You find that mean difference is $A-B=\Delta=0\pm0.001$

Now, we'd like to distinguish between two cases:

  1. The difference is really zero with high probability, for example, $\Delta=0\pm0.0001$
  2. Our tool is not sensitive enough to detect underlying non-zero difference

My argument would be that (1) and (2) are equivalent, but I have a feeling that there should be a statistics tool to tell them apart.

Additional detail: let's assume, that in next city where we measure A and B, the difference comes out as $\Delta=0.5\pm0.01$

  • 6
    $\begingroup$ $0\pm 0.0001$ is not "zero with high probability". $\endgroup$
    – Glen_b
    Jul 27, 2018 at 1:16
  • 2
    $\begingroup$ "The skunk did not spray my dog" is different from "I have no sense of smell." $\endgroup$
    – jbowman
    Jul 27, 2018 at 3:31
  • $\begingroup$ @Glen_b you are right, but I am trying to get better framework to think about it. Does it matter if in city nearby we measured $\Delta=0.5\pm0.01$? $\endgroup$ Jul 27, 2018 at 3:50
  • $\begingroup$ Sorry, I don't understand what you're asking -- does what matter? I was merely pointing out that a high probability that $A-B$ lies in a narrow interval around zero doesn't imply that $P(A-B=0)$ is close to $1$; indeed there's no obvious reason to think that it's larger than $0$. You may well want to dismiss this as quibbling, but such unclear thinking is no doubt part of why you're having trouble untangling things. A more clearly formulated questions will result in better answers. $\endgroup$
    – Glen_b
    Jul 27, 2018 at 6:36

1 Answer 1


What you are really talking about here is null hypothesis testing, although you express it in the form of a confidence interval. So, my answer will be in those terms.

First thing: If you truly measured your parameters in the population, then you don't use confidence intervals or null hypothesis testing. The difference between your populations is exactly what you measured.

However, assuming you meant you measured things in a sample from each population, then the reason Statement 1 and Statement 2 are not the same is that in Statement 1 you are talking about whether you can reject the null hypothesis. This depends on $\alpha$, which is the probability of making a Type I error (rejecting the null hypothesis when the null is, in fact, true). In Statement 2 you are talking about the Power of a statistical test. This depends on $\beta$, which is the probability of making a Type II error (the probability of failing to reject the null when in fact the null is false).

Suppose you have the null hypothesis $H_0: \Delta = 0$. If you get observed data where $p<\alpha$, then we call this a "significant effect", and we reject the null hypothesis. This does not tell us anything about the probability that $\Delta$ actually equals 0. What it tells us is that if $\Delta$ equals 0, then the probability of our data is small (equal to p). (If you want to know what the actual probability of the truth of the null hypothesis is, then you are going to need to move into the world of Bayesian statistics).

Power is defined as $1-\beta$. It is the conditional probability that you will reject your null hypothesis if the null hypothesis is, in fact, false. Power analysis depends on four factors:

1) The effect size

2) The $\alpha$-level

3) The power level you want ($1-\beta$)

4) The sample size

If you specify any 3 of these factors, you can calculate the fourth. So, if you want to know whether your test is powerful enough to detect a small effect size, you would enter the effect size, the sample size, and the $\alpha$, and you can calculate the probability that your test will detect that size effect.

A common use of power analysis is to determine sample size. In this case, you enter the $\alpha$, the effect size you want to be able to detect, and the probability you want to have to be able to detect that effect size. This will tell you what size sample you need.

G*Power is a free program that will allow you to do these analyses. There are lots of books, articles, and websites that cover the basics of power analysis.

  • $\begingroup$ thanks, i was thinking that Power is important term for my problem $\endgroup$ Jul 27, 2018 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.