# If p-value is exactly 1 (1.0000000), what should the confidence interval limits be to support null hypothesis being true? [closed]

This is purely a hypothetical question. A very common statement is that $$H_0$$ is never true, it's just a matter of sample size.

Let´s assume that for real there is absolutely no measurable difference between two means ($$\mu_1=\mu_2$$) drawn from normally distributed population (for both $$\mu=0$$ and $$\sigma$$ estimated $$=1$$). We assume $$N=16$$ per group and we use $$t$$-test. This would mean that $$p$$-value is $$1.00000$$ indicating that there is absolutely no discrepancy from $$H_0$$. This would indicate that test statistic is $$0$$. Mean difference between groups would be $$0$$. What would be the limits of $$95\%$$ confidence interval for the mean difference in this case? Would they be $$[0.0, 0.0]$$?

Main point in my question was that when can we really say that $$H_0$$ is true, i.e. $$\mu_1=\mu_2$$ in this case? Or when in frequentist framework we can truly say "no difference" when comparing two means?

• I'd say that this was already answered in here stats.stackexchange.com/questions/275677/…, but I'm not insisting on it.
– Tim
Sep 2, 2019 at 22:01
• I’m having trouble coming up with a way to get $p=1$ with positive population variances.
– Dave
Sep 2, 2019 at 22:35
• Possible duplicate of Relation between confidence interval and testing statistical hypothesis for t-test Sep 2, 2019 at 22:57
• "We assume N=16 per group and we use t-test. This would mean that p-value is 1.00000 indicating that there is absolutely no discrepancy from H0." Why do you argue that something (what does 'this' refer to?) means that the p-value is 1. Most typically the p-value is uniform distributed when H_0 is true, and p=1 happens almost never. Sep 3, 2019 at 7:37
• @MartijnWeterings Is absolutely correct - just because you're sampling two distributions that are in fact identical does not mean that you'll get a p-value of 1 when comparing them. By definition, 5% of the time you'll get a p-value below 0.05. Sep 3, 2019 at 14:34

A confidence interval for a t-test is of the form $$\bar{x}_1 - \bar{x}_2 \pm t_{\text{crit}, \alpha}s_{\bar{x}_1 - \bar{x}_2}$$, where $$\bar{x}_1$$ and $$\bar{x}_2$$ are the sample means, $$t_{\text{crit}, \alpha}$$ is the critical $$t$$ value at the given $$\alpha$$, and $$s_{\bar{x}_1 - \bar{x}_2}$$ is the standard error of the difference in means. If $$p=1.0$$, then $$\bar{x}_1 - \bar{x}_2 =0$$. So the formula is just $$\pm t_{\text{crit}, \alpha}s_{\bar{x}_1 - \bar{x}_2}$$, and the limits are just {$$-t_{\text{crit}, \alpha}s_{\bar{x}_1 - \bar{x}_2}$$, $$t_{\text{crit}, \alpha}s_{\bar{x}_1 - \bar{x}_2}$$}.

I'm not sure why you would think the limits would be $$\{0,0\}.$$ The critical $$t$$ value is not zero and the standard error of the mean difference is not zero.

Being super-lazy, using R to solve the problem numerically rather than doing the calculations by hand:

Define a function that will give normally distributed values with a mean of (almost!) exactly zero and a SD of exactly 1:

rn2 <- function(n) {r <- rnorm(n); c(scale(r)) }


Run a t-test:

t.test(rn2(16),rn2(16))

Welch Two Sample t-test

data:  rn2(16) and rn2(16)
t = 1.7173e-17, df = 30, p-value = 1
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.7220524  0.7220524
sample estimates:
mean of x    mean of y
6.938894e-18 8.673617e-19


The means are not exactly zero because of floating-point imprecision.

More directly, the CIs are $$\pm$$ sqrt(1/8)*qt(0.975,df=30); the variance of each mean is 1/16, so the pooled variance is 1/8.

# The CI can have any limits, but it is centered exactly around zero

For a two-sample T-test (testing for a difference in the means of two populations), a p-value of exactly one corresponds to the case where the observed sample means are exactly equal.$$^\dagger$$ (The sample variances can take on any values.) To see this, note that the p-value function for the test is is:

$$p \equiv p(\mathbf{x},\mathbf{y}) = \mathbb{P} \Bigg( \Bigg| \frac{\bar{X}-\bar{Y}}{\sqrt{S_Y/n_Y + S_Y/n_Y}} \Bigg| \geqslant \Bigg| \frac{\bar{x}-\bar{y}}{\sqrt{s_Y/n_Y + s_Y/n_Y}} \Bigg| \Bigg).$$

Thus, setting $$\bar{x}=\bar{y}$$ yields:

$$p(\mathbf{x},\mathbf{y}) = \mathbb{P} \Bigg( \Bigg| \frac{\bar{X}-\bar{Y}}{\sqrt{S_Y/n_Y + S_Y/n_Y}} \Bigg| \geqslant 0 \Bigg) = 1.$$

Now, suppose you form the standard (approximate) confidence interval using the Welch-Satterwaite approximation. In this case, assuming that $$\bar{x}=\bar{y}$$ (to give an exact p-value of one) gives the confidence interval:

$$\text{CI}(1-\alpha) = \Bigg[ 0 \pm \sqrt{\frac{s_X}{n_X} + t_{DF, \alpha/2} \frac{s_Y}{n_Y}} \Bigg],$$

where the degrees-of-freedom $$DF$$ is determined by the Welch-Satterwaite approximation. Depending on the observed sample variances in the problem, the confidence interval can be any finite interval centered around zero. That is, the confidence interval can have any limits, so long as it is centered exactly around zero.

$$^\dagger$$ Of course, if the underlying data actually come from a continuous distribution, this event occurs with probability zero, but let's assume it happens.

• The question says "σ estimated =1". Sep 3, 2019 at 21:49
• That condition is not necessary to get a p-value of one, so I have dropped it.
– Ben
Sep 4, 2019 at 0:15

It is difficult to have a cogent philosophical discussion about things that have 0 probability of happening. So I will show you some examples that relate to your question.

If you have two enormous independent samples from the same distribution, then both samples will still have some variability, the pooled 2-sample t statistic will be near, but not exactly 0, the P-value will be distributed as $$\mathsf{Unif}(0,1),$$ and the 95% confidence interval will be very short and centered very near $$0.$$

An example of one such dataset and t test:

set.seed(902)
x1 = rnorm(10^5, 100, 15)
x2 = rnorm(10^5, 100, 15)
t.test(x1, x2, var.eq=T)

Two Sample t-test

data:  x1 and x2
t = -0.41372, df = 2e+05, p-value = 0.6791
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.1591659  0.1036827
sample estimates:
mean of x mean of y
99.96403  99.99177


Here are summarized results from 10,000 such situations. First, the distribution of P-values.

set.seed(2019)
pv = replicate(10^4,
t.test(rnorm(10^5,100,15),rnorm(10^5,100,15),var.eq=T)$p.val) mean(pv)  0.5007066 # aprx 1/2 hist(pv, prob=T, col="skyblue2", main="Simulated P-values") curve(dunif(x), add=T, col="red", lwd=2, n=10001) Next the test statistic: set.seed(2019) # same seed as above, so same 10^4 datasets st = replicate(10^4, t.test(rnorm(10^5,100,15),rnorm(10^5,100,15),var.eq=T)$stat)
mean(st)
 0.002810332  # aprx 0
hist(st, prob=T, col="skyblue2", main="Simulated P-values")
curve(dt(x, df=2e+05), add=T, col="red", lwd=2, n=10001) And so on for the width of the CI.

set.seed(2019)
w.ci = replicate(10^4,
diff(t.test(rnorm(10^5,100,15),
rnorm(10^5,100,15),var.eq=T)\$conf.int))
mean(w.ci)
 0.2629603


It is almost impossible to get a P-value of unity doing an exact test with continuous data, where assumptions are met. So much so, that a wise statistician will ponder what might have gone wrong upon seeing a P-value of 1.

For example, you might give the software two identical large samples. The programming will carry on as if these are two independent samples, and give strange results. But even then the CI will not be of 0 width.

set.seed(902)
x1 = rnorm(10^5, 100, 15)
x2 = x1
t.test(x1, x2, var.eq=T)

Two Sample t-test

data:  x1 and x2
t = 0, df = 2e+05, p-value = 1
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.1316593  0.1316593
sample estimates:
mean of x mean of y
99.96403  99.96403

• That's all fair enough, however, because the normal distribution is continuous, the probability for any specific example is zero, doesn't matter whether u1=u2 or u1-u2=-0.977 or whatever. I was tempted to comment along the lines of "this will never happen and chances are something wnet wrong in that case" as well, but then I thought, no, it makes some sense to say, assume this has happened, accepting that this has probability zero as any specific example. Sep 3, 2019 at 11:02
• This is the right answer to the wrong question Sep 3, 2019 at 11:08
• @ David. Possibly so. If you can state what you believe to be the right question and suggest an answer, that might be helpful all around. I attempted only to address a few of what I thought were several misconceptions. Sep 3, 2019 at 18:22
• The OP stated " A very common statement is that H0 is never true." @BruceET's answer demonstrates WHY H0 can never be accepted. The closer H0 comes to being true, the more uniformly random P becomes, that means a P between (0.98,0.99) is just as likely as a P between (0.1, 0.2) when H0 is true. Sep 4, 2019 at 19:14

The straightforward answer (+1 to Noah) will explain that the confidence interval for the mean difference may still be of nonzero length because it depends on the observed variation in the sample in a different way than the p-value does.

However you might still wonder why it is like that. Since it is not soo strange to imagine that a high p-value also means a small confidence interval. After all they both correspond to something that is close to a confirmation of the null hypothesis. So why is this thought not correct?

A high p-value is not the same as a small confidence interval.

• The p-value is an indicator of how extreme a particular observation is (extreme given some hypothesis) by expressing how probable it is to observe a given deviation. It is an expression of the observed effect size in relation to the accuracy of the experiment (a large observed effect size might not mean very much when the experiment is such 'inaccurate' that these observations are not extreme from a statistical/probabilistic point of view). When you observe a p-value of 1 then this (only) means that you observed zero effect because the probability to observe such zero result or larger is equal to 1 (but this is not the same as that there is zero effect).

Sidenote: Why p-values? The p-value expresses the actual observed effect size in relation to the expected effect sizes (probabilities). This is relevant because experiments might, by design, generate observations of some relevant effect size by pure chance due to common fluctuations in data/onservations. Requiring that an observation/experiment has a low p-value means that the experiment has a high precision - that is: the observed effect size is less often/likely due to chance/fluctuations (and it might be likely due to a true effect).

Sidenote: for continuous variables this p-value equal to 1 occurs almost never because it is an event that has zero measure (E.g. for a normal distributed variable $$X\sim N (0,1)$$ you have $$\mathbb {P}(X=0)=0$$). But for a discrete variable or discretized continuous variable it can be the case (at least the probability is nonzero).

• The confidence interval might be seen as the range of values for which an $$\alpha$$ level hypothesis test would succeed (for which the p-value is above $$\alpha$$).

You should note that a high p-value is not (neccesarily) a proof/support/whatever for the null hypothesis. The high p-value only means that the observation is not remarkable/extreme for a given null hypothesis, but this might just as well be the case for the alternative hypothesis (ie the result is in accordance with both hypotheses yes/no effect). This typically occurs when the data does not carry much information (eg high noise or small sample).

Example: Imagine you have a bag of coins for which you have fair and unfair coins and you want to classify a certain coin by flipping it 20 times. (say the coin is a bernoulli variable with $$p \approx 0.5$$ for fair coins and $$p \sim U (0,1)$$ for unfair coins. In this case, when you observe 10 heads and 10 tails, then you might say the p-value is equal to 1, but I guess that it is obvious that an unfair coin might just as well create this result and we should not rule out the possibility that the coin is unfair.

Main point in my question was that when can we really say that $$H_0$$ is true, i.e. $$\mu_1=\mu_2$$ in this case?

No, because "absence of evidence is not evidence of absence." Probability can be thought as an extension of logic, with added uncertainties, so imagine for a moment that instead of real numbers on unit interval, the hypothesis test would return only the binary values: 0 (false) or 1 (true). In such case, the basic rules of logic apply, as in the following example:

• If it rained outside, then the ground being wet is likely.
• The ground is wet.
• Therefore, it rained outside.

The ground could very well be wet because it rained. Or it could be due to a sprinkler, someone cleaning their gutters, a water main broke, etc. More extreme examples can be found in the link above.

As about confidence interval, if your sample is large, and $$\mu_1 - \mu_2 \to 0$$, then the confidence interval for the difference would become extremely narrow, but non-zero. As noticed by others, you could observe things like exact ones and zeros, but rather because of the floating-point precision limitations.

Even if you observed $$p = 1$$ and the $$\pm 0$$ confidence interval, you still need to keep in mind that the test gives you only the approximate answer. When doing hypothesis testing, we not only make the assumption that $$H_0$$ is true, but also make a number of other assumptions, like that the samples are independent and come from normal distribution, what is never the case for real-world data. The test gives you an approximate answer, to ill-posed question, so it cannot "prove" the hypothesis, it can just say "under those unreasonable assumptions, this would be unlikely".

Nothing stops you from using standard t- or Gauss-formulae for computing the confidence interval - all informations needed are given in your question. p=1 doesn't mean that there's anything wrong with that. Note that p=1 does not mean that you can be particularly sure that the H0 is true. Random variation is still present and if u0=u1 can happen under the H0, it can also happen if the true value of u0 is slightly different from the true u1, so there will be more in the confidence interval than just equality.

• I did some editing, I hope it's more defined now. Sep 2, 2019 at 22:26
• OK, I removed references to what was ill-defined in the earlier version. The question has in the meantime been answered properly by others. Sep 3, 2019 at 10:59
• Please use MathJax notation Sep 3, 2019 at 11:07

A very common statement is that H0 is never true, it's just a matter of sample size.

Not among people who know what they're talking about, and are speaking precisely. Traditional hypothesis testing never concludes that the null is true, but whether the null is true or not is separate from whether the null is concluded to be true.

This would mean that p-value is 1.00000

For a two-tailed test, yes.

indicating that there is absolutely no discrepancy from H0.

$$H_0$$ is a statement about the distribution. The mode of the distribution given in $$H_0$$ is $$0$$, so there's no discrepancy between the observation and the mode of the distribution, but it's not quite correct to say there's no discrepancy from $$H_0$$. No individual result would be a discrepancy, because any value could come from the distribution. Each p-value is equally likely. Getting a p-value of exactly .01 is just as likely as getting a p-value of exactly 1 (apart from discretization issues). If you had a bunch of independent samples, and their distribution didn't match what $$H_0$$ predicts, that would much more legitimately be called a "discrepancy" than would merely seeing a single sample whose mean doesn't match the mode.

What would be the limits of 95% confidence interval for the mean difference in this case?

To first approximation, the limits of a 95% confidence interval are about twice the applicable standard deviation. There is no discontinuity at zero. If you find a function $$f(\epsilon)$$ that finds the 95% confidence interval for a difference in means of $$\epsilon$$, you can simply take $$\lim_{\epsilon \rightarrow 0}f(\epsilon)$$ to find the confidence interval for a mean difference of zero.

Main point in my question was that when can we really say that H0 is true, i.e. μ1=μ2 in this case?

We can say whatever we want. However, saying that a test shows the null to be true is not consistent with traditional hypothesis testing, regardless of the results. And doing so is not well-founded from an evidenciary standpoint. The alternative hypothesis, that the means are not the same, encompasses all possible difference in means. The alternative hypothesis is "The difference in means is $$1$$, or $$2$$, or $$3$$, or $$.5$$, or $$.1$$, ..." We can posit an arbitrarily small difference in means, and that will be consistent with the alternative hypothesis. And with an arbitrarily small difference, the probability given that mean is arbitrarily close to the probability given the null. Also, the alternative hypothesis encompasses not only the possibility that the parameters of the distributions, such as the mean, are different, but that there's an entirely different distribution. For instance, the alternative hypothesis encompasses "The two samples will always have a difference in means that this is either exactly 1 or exactly 0, with probability .5 for each". The results are more consistent with that then they are with the null.