# How to know if the p value will increase or decrease

you conduct a hypothesis test. After 100 sample observations you observe values for sample mean and sample standard deviation that do not lead to the rejection of null hypothesis since p value is 0.12. After observing 200 sample observations you still observe the same sample mean and standard deviation. What will happen to the p value?

a. increase
b. decrease
c. stay the same
d. may increase or decrease

I got this question in an interview and I am so confused. Can someone please help me answer this. I think the answer should be d, but I am unsure.

• The intuition here is that having a larger sample size should make you more confident that your sample mean reflects reality, which would correspond to a smaller p-value. Mar 21, 2021 at 12:21

I believe that more precision could be added to exactly solve the problem and know what test are we talking about. But because you are talking about 1 sample mean and 1 standard deviation, I will assume a classic Z-test Statistics.

You are trying to see if the average of your sample $$\bar{x}$$ is significantly different from $$\mu$$. The "precision of your average" is given by the standard deviation expressed as: $$\sigma/\sqrt{n}$$. All of this can be found here https://en.wikipedia.org/wiki/Z-test

One can see that the more sample you add, the more your estimated variance will shrink. This in turns means a higher value for Z and thus a lower p-value, according to the following formula:

$$Z = \frac{(\bar{x}-\mu)}{(\sigma/\sqrt{n})}$$

A more intuitive way of seeing it, is that the more you draw sample, the more confident you are about the average because the smaller is standard deviation around that average. Now there will be a point, where that average is "precise enough" to be significantly different than any number $$\mu$$ that you were trying to compare it to.

In the case you were given, the p-value would then decrease.

 Thanks for the precision from Sal Mangiafico : in the case where $$\bar{x} =\mu$$ then the p-value will remain unchanged and equal to 1

• I do think this is the point of the question. One outside case is that if the sample mean is exactly equal to mu (for both samples described), then the p-value would be exactly 1 for both samples. Mar 21, 2021 at 13:28
• +1 answer. Essentially, as n increases the difference $\bar x - \mu_0$ increases and the variance decreases, making $Z$ larger (if $\mu_0 \ne \mu$). However, the part with the average being significantly different than any $\mu_0$ only holds if $\mu_0$ is different than the true value $\mu$ (i.e. $H0: \mu \ne \mu_0$). Mar 21, 2021 at 14:31

I thought I would add some code so OP can easily see this effect in action. This answer supports that by @Romain, except I used a one-sample t-test instead of a one-sample Z-test. That won't make much of a difference.

A is a sample of 100 observations with a mean of approximately 10 and a standard deviation of approximately 3. (These could be changed in the rnorm function.)

B will be a similar sample of 200 observations. A and B have exactly the same mean. Their sample standard deviation will be just slightly different, because of the way sample standard deviation is calculated.

This code can be run in R or at rdrr.io/snippets. The code is a little complex, but the output is easy to read. You can run it many times to see the behavior of the p-value starting with different samples.

Funny = function(){
A   = rnorm(100, 10, 3)
lA  = length(A)
mA  = mean(A)
sdA = sd(A)
pA  = t.test(A, mu=10)$p.value B = c(A,A) lB = length(B) mB = mean(B) sdB = sd(B) pB = t.test(B, mu=10)$p.value

Out = data.frame(Statistic=rep("A", 8), Value = rep(0.00, 8))
Out$$Statistic = c("n for A", "Mean of A", "Standard deviation of A", "t-test p-value for A", "n for B", "Mean of B", "Standard deviation of B", "t-test p-value for B") Out$$Value = c(signif(lA,3), signif(mA,3), signif(sdA,3), signif(pA,3),
signif(lB,3), signif(mB,3), signif(sdB,3), signif(pB,3))

return(Out)
}

Funny()

### Example output
###
###                 Statistic  Value
### 1                 n for A 100.000
### 2               Mean of A 10.200
### 3 Standard deviation of A  3.140
### 4    t-test p-value for A  0.518
### 5                 n for B 200.000
### 6               Mean of B 10.200
### 7 Standard deviation of B  3.140
### 8    t-test p-value for B  0.358