# How to perform t-test with huge samples?

I have two populations, One with N=38,704 (number of observations) and other with N=1,313,662. These data sets have ~25 variables, all continuous. I took mean of each in each data set and computed the test statistic using the formula

t=mean difference/std error

The problem is of the degree of freedom. By formula of df=N1+N2-2 we'll have more freedom than the table can handle. Any suggestions on this? How to check the t statistic here. I know that the t-test is used for handling samples but what if we apply this on large samples.

chl already mentioned the trap of multiple comparisons when conducting simultaneously 25 tests with the same data set. An easy way to handle that is to adjust the p value threshold by dividing them by the number of tests (in this case 25). The more precise formula is: Adjusted p value = 1 - (1 - p value)^(1/n). However, the two different formulas derive almost the same adjusted p value.

There is another major issue with your hypothesis testing exercise. You will most certainly run into a Type I error (false positive) whereby you will uncover some really trivial differences that are extremely significant at the 99.9999% level. This is because when you deal with a sample of such a large size (n = 1,313,662), you will get a standard error that is very close to 0. That's because the square root of 1,313,662 = 1,146. So, you will divide the standard deviation by 1,146. In short, you will capture minute differences that may be completely immaterial.

I would suggest you move away from this hypothesis testing framework and instead conduct an Effect Size type analysis. Within this framework the measure of statistical distance is the standard deviation. Unlike the standard error, the standard deviation is not artificially shrunk by the size of the sample. And, this approach will give you a better sense of the material differences between your data sets. Effect Size is also much more focused on confidence interval around the mean average difference which is much more informative than the hypothesis testing focus on statistical significance that often is not significant at all. Hope that helps.

• +1 for bringing out the key ideas: (1) we can guarantee the means will differ when the datasets are this large and (2) some other analysis is likely to be more appropriate and useful. But because we don't know about the purpose of the analysis, we should be cautious about making specific recommendations.
– whuber
Oct 30 '10 at 20:45
• Thanks Gaetan..got you..I think what I take away from this is that standard deviation is a better measure when you have large samples like mine..please let me know if I missed anything. Nov 1 '10 at 10:43
• ayush... You are right. That is basically it. And, this is because your standard error will become so small (due to the large sample size). This in turn overstates the statistical distance between your test and control groups. And, causes you to ultimately run into a Type I Error (uncover a difference that is so small as to be immaterial). This is a common problem in hypothesis testing with large samples. Nov 1 '10 at 16:17

Student's t-distribution becomes closer and closer the the standard normal distribution as the degrees of freedom get larger. With 1313662 + 38704 – 2 = 1352364 degrees of freedom, the t-distribution will be indistinguishable from the standard normal distribution, as can be seen in the picture below (unless perhaps you're in the very extreme tails and you're interested in distinguishing absolutely tiny p-values from even tinier ones). So you can use the table for the standard normal distribution instead of the table for the t-distribution.

• Guys,thanks for the answer. I have a data to analyse. How do i attach data to this. Lots to ask you people..Thanks in anticipation. Expecting a prompt reply. Oct 30 '10 at 10:18
• Huh? You said in the question you'd already computed the t-statistic, and chl has provided sample R code. What more do you want? By the way, I'm not sure you have any right to expect or request a prompt reply; we don't get paid for this you know. Oct 30 '10 at 10:45
– chl
Oct 30 '10 at 13:01
• @ayush Ah, and I just realize that you never vote up any of the answers that were provided to you (though you have enough rep now).
– chl
Oct 30 '10 at 15:31
• @chl-- yeah..even I realize this fault of mine and shall rectify this for sure in posts to come..Thanks for pointing this out..Consider me for some days a naive amateur.. Nov 1 '10 at 10:27

The $t$ distribution tend to the $z$ (gaussian) distribution when $n$ is large (in fact, when $n>30$, they are almost identical, see the picture provided by @onestop). In your case, I would say that $n$ is VERY large, so that you can just use a $z$-test. As a consequence of the sample size, any VERY small differences will be declared significant. So, it is worth asking yourself if these tests (with the full data set) are really interesting.

Just to be sure, as your data set includes 25 variables, you are making 25 tests? If this is the case, you probably need to correct for multiple comparisons so as not to inflate the type I error rate (see related thread on this site).

BTW, the R software would gives you the p-values you are looking for, no need to rely on Tables:

> x1 <- rnorm(n=38704)
> x2 <- rnorm(n=1313662, mean=.1)
> t.test(x1, x2, var.equal=TRUE)

Two Sample t-test

data:  x1 and x2
t = -17.9156, df = 1352364, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.1024183 -0.0822190
sample estimates:
mean of x   mean of y
0.007137404 0.099456039


You can use the following python function which I wrote, that can calculate the size effect. The test is straightforward here

import numpy as np
from scipy.stats import t

def Independent_tTest(x1, x2, std1, std2, n1, n2):
'''Independent t-test between two sample groups

Note:
The test assumptions:
H0: The two samples are not significantly different (from same population)
H1: The two samples are siginficantly different (from two populations)
- Accept the H1 if t-value > t-critical or p-value value < p-value critical
Args:
x1(float): mean of the first sample group.
x2(float): mean of the second sample group.
std1(float): standard deviation of first sample group.
std2(float): standard devation of second sample group.

Return:
degree_of_freedome, t-statistics, p-value

'''
degree_of_freedom = n1 + n2  -2
corrected_degree_of_freedom = (((std1**2/n1) + (std2**2/n2))**2)/(((std1**4)/((n1**2)*(n1-1)))+((std2**4)/((n2**2)*(n2-1))))

poolvar = ((n1-1)*(std1**2)+ (n2-1)*(std2**2))/corrected_degree_of_freedom
t_value = (x1 -x2)/np.sqrt(poolvar*((1/n1)+ (1/n2)))
sig = 2 * (1-(t.cdf(abs(t_value), corrected_degree_of_freedom)))
effect_size = np.sqrt((t_value**2)/(t_value**2+corrected_degree_of_freedom))
return f"corrected degree of freedom {corrected_degree_of_freedom:0.4f} give a t-value = {t_value:0.4f}, with significant = {sig:0.4f} with effectsize ={effect_size:0.4f}"

• Although implementation is often mixed with substantive content in questions, we are supposed to be a site for providing information about statistics, machine learning, etc., not code. It can be good to provide code as well, but please elaborate your substantive answer in text for people who don't read this language well enough to recognize & extract the answer from the code. May 1 '20 at 13:06