I am trying to answer this question:

There is no difference between two population means. From each of these two populations
a sample is selected and then for both samples the means are calculated. 

Consider this:
"since the null hypo is true, a significant result can never be found when we perform
a t-test for independent measures (two-tailed, alpha = 0.05)

Can we say that is statement is in/correct, partially correct, or insufficient to answer? I am really unsure of which path to take.


1 Answer 1


This is totally false. Let's do an R simulation.

ps <- rep(NA, 1000)
for (i in 1:1000){

    # Simulate data from two identical distributions
    x <- rnorm(100)
    y <- rnorm(100)

    # Test the means, save the p-value
    ps[i] <- t.test(x, y, var.equal=T)$p.value


When you run this, you will notice that a number of the p-values fall below $0.05$. In fact, about $5\%$ will fall below $0.05$.

It turns out that, under the null hypothesis, the p-value has a uniform distribution on $[0,1]$.

  • $\begingroup$ You can prove a far stronger and far-reaching statement: no matter what the populations may be, provided only that at least one of them is not constant, and no matter what significance level you choose, the chance of finding a significant difference (using a t test) is always positive. $\endgroup$
    – whuber
    Jan 14, 2021 at 21:25
  • $\begingroup$ I really appreciate both of your comments and effort in helping me out. I sincerely thank you both. $\endgroup$
    – e. erhan
    Jan 14, 2021 at 21:34
  • $\begingroup$ @whuber, can I ask what you meant by being not constant for a population? Also, should I understand that even if these two populations not only had the same mean but also the same standard deviation, there is still a positive chance of finding a significant difference? $\endgroup$
    – e. erhan
    Jan 14, 2021 at 21:53
  • 1
    $\begingroup$ @e.erhan The example I gave uses identical distributions: same mean, same standard deviation, same everything. $\text{//}$ A constant population is a population where every observation has the same value. $\endgroup$
    – Dave
    Jan 14, 2021 at 21:55
  • $\begingroup$ @Dave Thanks for the clarification! $\endgroup$
    – e. erhan
    Jan 14, 2021 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.