Large sample size for t-test

Question

I am working on a physics experiment, and I ran 500 runs to calculate a parameter. I want to use the t-test to see if my mean from 500 runs matches the actual value.

• u0: mean is actual value
• u1: mean is not actual value

When I tried with 100 data first, it returned a high p-value which did not reject the null hypothesis. So I was very happy with my results. However, when I tried with 500 data, it provided me low p-value (<0.05). So I have to reject my hypothesis.

I find this very strange because my mean from 100 data and 500 data are very close. I believe the higher sample size changed that insignificant difference to significant difference.

So what sample size is recommended? Any article that I can look for?

Answer 1

As has been pointed out already, essentially you have formulated a hypothesis that does not test what you wish to actually show. Your interpretation of the $p$-value also implies this.

If your intended goal is to show with a high degree of confidence that an empirically estimated statistic is "equivalent" to a theoretically derived parameter, then one correct form of a hypothesis test to establish equivalence might be $$H_0 : |\hat \mu - \mu| \ge \Delta \quad {\mathrm{vs.}} \quad H_a : |\hat \mu - \mu| < \Delta,$$ where $\mu$ is the theoretical mean, $\hat \mu$ is your estimator of $\mu$, and $\Delta > 0$ is some chosen value that represents your belief of how close these two values must be in order to consider them "equivalent."

This test is a composite hypothesis, which needs to be written as the intersection of two one-sided tests: $$\begin{align*} H_{01} : \hat\mu - \mu \ge \Delta \quad &{\mathrm{vs.}} \quad H_{a1} : \hat\mu - \mu < \Delta, \\ H_{02} : \hat\mu - \mu \le -\Delta \quad &{\mathrm{vs.}} \quad H_{a2} : \hat\mu - \mu > -\Delta. \end{align*}$$ Both $H_{01}$ and $H_{02}$ must be rejected in order to accept $H_a$. How you proceed from here depends on your assumptions regarding the sampling distribution of the estimator $\hat\mu$. If you assume it is normal with known variance, then you have two one-sided $z$-tests; if you assume it is normal with unknown variance, then you have two one-sided $t$-tests; and so forth. Then you want to determine the form of the test statistic, and calculate a $p$-value for this test.

If the $p$-value is sufficiently small, then you will reject $H_0$ and say that the measured mean is "equivalent" to the theoretical mean. If not, the decision is to fail to reject the null and say that the data gathered contains insufficient evidence to suggest the empirical mean equals the theoretical mean.

Again, I remind you that the notion of "equivalence" or "equality" is necessarily defined by your choice of $\Delta$. You cannot choose $\Delta = 0$ because the critical region for such a test is the empty set, and such a test will have no power--you will always fail to reject the null.

Finally, this hypothesis is not the only one possible. Depending on your beliefs about the nature of the sampling distribution of your estimator, other hypotheses may be more appropriate.

Answer 2

It is not strange, because the size $\alpha$ of your test should depend on the sample size: "in large samples it is more appropriate to choose a size of 1% or less rather than the 'traditional' 5%. Similarly, in very small samples we may prefer to work with a significance level of 10%" (Marno Verbeek, A Guide to Modern Econometrics, §2.5.7).

See also: M. Lin, H.C. Lucas and G. Shmueli, "Too Big to Fail: Large Samples and the p-Value Problem".

But one can also say that the sample size depends on the (desired) size $\alpha$ -- and on the (desired) power $1-\beta$. This is called "planning of sample size with power approach", i.e. controlling the risks of making Type I and Type II errors. See Kutner et al., Applied Linear Statistical Models, §16.10; Donner, Approaches to Sample Size Estimation in the Design of Clinical Trials - A Review; Kadam and Bhalerao, Sample size calculation; Dell, Holleran and Ramakrishnan, Sample Size Determination, etc.

There is also a "planning of sample size with estimation approach", which is used to control the precision of estimates of important effects (like your parameter, I guess). The essence of the approach is to specify the major comparisons of interest and to determine the expected widths of the confidence intervals for various sample sizes (Kutner et al., §17.8). See also Maxwell, Kelley and Raush, Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation.

In general, $\uparrow n \Leftrightarrow\; \downarrow \alpha$, thus you should not use $\alpha=0.05$ for whatever sample size, but you should look for an appropriate sample size for a given $\alpha$ -- as you do when you wonder about what sample size is recommended. You can also look at Wikipedia and this NIST page.

EDIT: Please, let me add a few remarks. The point estimate is just a 'best guess', the standard error is a measure of precision, but the confidence interval gives us the range of values consistent with the data. Look at this example:

> set.seed(1235321)
> x <- runif(300, -2, 2)
> eps <- rnorm(300)
> y <- 1 + 3 * x[1:3] + eps[1:3]
> fit.3 <- lm(y ~ x[1:3])
> y <- 1 + 3 * x[1:30] + eps[1:30]
> fit.30 <- lm(y ~ x[1:30])
> y <- 1 + 3 * x + eps
> fit.300 <- lm(y ~ x)
> summary(fit.3)$coefficients
            Estimate Std. Error  t value  Pr(>|t|)
(Intercept) 1.090149  1.0577209 1.030658 0.4903893
x[1:3]      3.025560  0.9668152 3.129409 0.1969026  
> summary(fit.30)$coefficients
            Estimate Std. Error   t value     Pr(>|t|)
(Intercept) 1.219546  0.1745157  6.988173 1.340782e-07
x[1:30]     2.935391  0.1444060 20.327353 2.648767e-18
> summary(fit.300)$coefficients
            Estimate Std. Error  t value      Pr(>|t|)
(Intercept) 1.014186 0.05521139 18.36915  6.341771e-51
x           2.976312 0.04684994 63.52863 2.775195e-175

You can see that, as the sample size gets larger, standard errors get smaller and $t$ values get larger. Thus you get 'more significant' point estimates (look at p-values) even if the data generation process is exactlty the same. Your degree of belief (i.e. $\alpha$) depends on sample size. But look at the confindence intervals:

> confint(fit.3)
                 2.5 %   97.5 %
(Intercept) -12.349469 14.52977
x[1:3]       -9.258991 15.31011
> confint(fit.30)
                2.5 %   97.5 %
(Intercept) 0.8620666 1.577025
x[1:30]     2.6395885 3.231193
> confint(fit.300)
                2.5 %   97.5 %
(Intercept) 0.9055328 1.122840
x           2.8841137 3.068511

Now you can see that each confidence interval contains the 'true value' of the coefficient for $x$, and that means that $3$ (the true value) is consistent with the data even when (the sample size is very small, therefore) the p-value is large. This is why confidence intervals are often preferred to $t$-tests and p-values ("If the confidence interval embraces too broad a set of values for $\theta$, then the dataset is not sufficiently informative to render inferences about $\theta$. On the other hand if the confidence interval is tight, then the data have produced an accurate estimate, and the focus should be on the value and interpretation of this estimate. In contrast, the statement 'the t-ratio is highly significant' has little interpretive value", Hansen, Econometrics, §8.8).

As to the proper interpretation of confidence intervals, I'd suggest Tom Siegfried, Scientists’ grasp of confidence intervals doesn’t inspire confidence, and, of course, J. Neyman, Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability, pages 347-349 (cited by T. Siegfried).

Answer 3

It's not strange at all. Having a larger sample shrinks your standard errors, improving your ability to reject the null when the sample mean is only slightly different from it. This should be your conclusion: that the sample mean is slightly different from the null hypothesis. Maybe you should worry more about the effect size. In any case, the recommended sample size is "as large as it can get."