I would like to know the relationship between sample size and p-value.

When comparing two independent means.

  1. Sample size calculation

1) Expected difference between two means: 2.8

2) Expected standard deviation: 6.25

3) Desired power(1-beta): 90%

4) Level of significance(alpha): 0.05

Two sided, Equal group sizes

=> By the typical formula calculation, the required sample size of each group is 105

  1. P-value calculation

Imagine that all real data are exactly same with the assumption when it used in sample size calculation.

Group 1

Mean: 0

Standard deviation: 6.25

Sample size: 105

Group 2

Mean: 2.8

Standard deviation: 6.25

Sample size: 105

=> By the typical formula calculation, I got it below;

Difference: 2.8

Standard error: 0.863

95% CI: 1.0995 to 4.5005

t-statistic: 3.246

DF: 208

Significance level: P=0.0014

My question(or maybe my misunderstanding) is why P-value(0.0014) is not calculated 0.05.

In other words,

I estimated mean difference is 2.8 and the real data is same 2.8.

I estimated standard deviation is 6.25 and the real data is same 6.25.

But why not the calculated p-value is not 0.05?

It would be very appreciated any reply for this.

  • $\begingroup$ How have you come to your estimation? Why would it be exactly 0.05? $\endgroup$ – william3031 Sep 16 '19 at 6:03
  • $\begingroup$ The procedure for the sample size asks you to supply significance level, power, variance of data, and size of difference to be detected. Suppose you say significance level 5%. Then if $H_0$ true, you can expect P-value to be below 0.5 (rejection) 5% of the time you run the expt, but no particular P-value is 'guaranteed'. $\endgroup$ – BruceET Sep 16 '19 at 9:18

Using some normally distributed data as per your description form the function described here: https://stackoverflow.com/questions/18919091/generate-random-numbers-with-fixed-mean-and-sd

rnorm2 <- function(n,mean,sd) { mean+sd*scale(rnorm(n)) }
g1 <- rnorm2(105, 0, 6.25)
g2 <- rnorm2(105, 2.8, 6.25)
t.test(g1, g2)

This produces:

    Welch Two Sample t-test

data:  g1 and g2
t = -3.2461, df = 208, p-value = 0.001364
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -4.500524 -1.099476
sample estimates:
   mean of x    mean of y 
8.208546e-17 2.800000e+00 

The difference in means is not equal to zero. The p-value here is less than 0.05

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.