I have two populations, One with N=38,704 (number of observations) and other with N=1,313,662. These data sets have ~25 variables, all continuous. I took mean of each in each data set and computed the test statistic using the formula

t=mean difference/std error

The problem is of the degree of freedom. By formula of df=N1+N2-2 we'll have more freedom than the table can handle. Any suggestions on this? How to check the t statistic here. I know that the t-test is used for handling samples but what if we apply this on large samples.


4 Answers 4


chl already mentioned the trap of multiple comparisons when conducting simultaneously 25 tests with the same data set. An easy way to handle that is to adjust the p value threshold by dividing them by the number of tests (in this case 25). The more precise formula is: Adjusted p value = 1 - (1 - p value)^(1/n). However, the two different formulas derive almost the same adjusted p value.

There is another major issue with your hypothesis testing exercise. You will most certainly run into a Type I error (false positive) whereby you will uncover some really trivial differences that are extremely significant at the 99.9999% level. This is because when you deal with a sample of such a large size (n = 1,313,662), you will get a standard error that is very close to 0. That's because the square root of 1,313,662 = 1,146. So, you will divide the standard deviation by 1,146. In short, you will capture minute differences that may be completely immaterial.

I would suggest you move away from this hypothesis testing framework and instead conduct an Effect Size type analysis. Within this framework the measure of statistical distance is the standard deviation. Unlike the standard error, the standard deviation is not artificially shrunk by the size of the sample. And, this approach will give you a better sense of the material differences between your data sets. Effect Size is also much more focused on confidence interval around the mean average difference which is much more informative than the hypothesis testing focus on statistical significance that often is not significant at all. Hope that helps.

  • 9
    $\begingroup$ +1 for bringing out the key ideas: (1) we can guarantee the means will differ when the datasets are this large and (2) some other analysis is likely to be more appropriate and useful. But because we don't know about the purpose of the analysis, we should be cautious about making specific recommendations. $\endgroup$
    – whuber
    Oct 30, 2010 at 20:45
  • $\begingroup$ Thanks Gaetan..got you..I think what I take away from this is that standard deviation is a better measure when you have large samples like mine..please let me know if I missed anything. $\endgroup$ Nov 1, 2010 at 10:43
  • 1
    $\begingroup$ ayush... You are right. That is basically it. And, this is because your standard error will become so small (due to the large sample size). This in turn overstates the statistical distance between your test and control groups. And, causes you to ultimately run into a Type I Error (uncover a difference that is so small as to be immaterial). This is a common problem in hypothesis testing with large samples. $\endgroup$
    – Sympa
    Nov 1, 2010 at 16:17

Student's t-distribution becomes closer and closer the the standard normal distribution as the degrees of freedom get larger. With 1313662 + 38704 – 2 = 1352364 degrees of freedom, the t-distribution will be indistinguishable from the standard normal distribution, as can be seen in the picture below (unless perhaps you're in the very extreme tails and you're interested in distinguishing absolutely tiny p-values from even tinier ones). So you can use the table for the standard normal distribution instead of the table for the t-distribution.

alt text

  • $\begingroup$ Guys,thanks for the answer. I have a data to analyse. How do i attach data to this. Lots to ask you people..Thanks in anticipation. Expecting a prompt reply. $\endgroup$ Oct 30, 2010 at 10:18
  • 5
    $\begingroup$ Huh? You said in the question you'd already computed the t-statistic, and chl has provided sample R code. What more do you want? By the way, I'm not sure you have any right to expect or request a prompt reply; we don't get paid for this you know. $\endgroup$
    – onestop
    Oct 30, 2010 at 10:45
  • 1
    $\begingroup$ @ayush For your preceding question, I provide a complete answer to your question (IMHO) -- then I gave some follow-up to your comments before stopping when I thought you were asking for another question which is not the purpose of comment option here. So, I would suggest that either you clearly state if your question relates to theoretical consideration or applied data analysis (in the latter case, give us a reproducible example) or separate your questions. BTW, you still have the option to accept answers that you find useful (again, wrt. your original question, not the comments that follow). $\endgroup$
    – chl
    Oct 30, 2010 at 13:01
  • 2
    $\begingroup$ @ayush Ah, and I just realize that you never vote up any of the answers that were provided to you (though you have enough rep now). $\endgroup$
    – chl
    Oct 30, 2010 at 15:31
  • $\begingroup$ @chl-- yeah..even I realize this fault of mine and shall rectify this for sure in posts to come..Thanks for pointing this out..Consider me for some days a naive amateur.. $\endgroup$ Nov 1, 2010 at 10:27

The $t$ distribution tend to the $z$ (gaussian) distribution when $n$ is large (in fact, when $n>30$, they are almost identical, see the picture provided by @onestop). In your case, I would say that $n$ is VERY large, so that you can just use a $z$-test. As a consequence of the sample size, any VERY small differences will be declared significant. So, it is worth asking yourself if these tests (with the full data set) are really interesting.

Just to be sure, as your data set includes 25 variables, you are making 25 tests? If this is the case, you probably need to correct for multiple comparisons so as not to inflate the type I error rate (see related thread on this site).

BTW, the R software would gives you the p-values you are looking for, no need to rely on Tables:

> x1 <- rnorm(n=38704)
> x2 <- rnorm(n=1313662, mean=.1)
> t.test(x1, x2, var.equal=TRUE)

    Two Sample t-test

data:  x1 and x2 
t = -17.9156, df = 1352364, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -0.1024183 -0.0822190 
sample estimates:
  mean of x   mean of y 
0.007137404 0.099456039 

You can use the following python function which I wrote, that can calculate the size effect. The test is straightforward here

import numpy as np 
from scipy.stats import t

def Independent_tTest(x1, x2, std1, std2, n1, n2): 
    '''Independent t-test between two sample groups

        The test assumptions:
            H0: The two samples are not significantly different (from same population)
            H1: The two samples are siginficantly different (from two populations)
            - Accept the H1 if t-value > t-critical or p-value value < p-value critical
        x1(float): mean of the first sample group.
        x2(float): mean of the second sample group.
        std1(float): standard deviation of first sample group.
        std2(float): standard devation of second sample group.

        degree_of_freedome, t-statistics, p-value

    degree_of_freedom = n1 + n2  -2
    corrected_degree_of_freedom = (((std1**2/n1) + (std2**2/n2))**2)/(((std1**4)/((n1**2)*(n1-1)))+((std2**4)/((n2**2)*(n2-1))))

    poolvar = ((n1-1)*(std1**2)+ (n2-1)*(std2**2))/corrected_degree_of_freedom
    t_value = (x1 -x2)/np.sqrt(poolvar*((1/n1)+ (1/n2)))
    sig = 2 * (1-(t.cdf(abs(t_value), corrected_degree_of_freedom)))
    effect_size = np.sqrt((t_value**2)/(t_value**2+corrected_degree_of_freedom))
    return f"corrected degree of freedom {corrected_degree_of_freedom:0.4f} give a t-value = {t_value:0.4f}, with significant = {sig:0.4f} with effectsize ={effect_size:0.4f}"
  • 1
    $\begingroup$ Although implementation is often mixed with substantive content in questions, we are supposed to be a site for providing information about statistics, machine learning, etc., not code. It can be good to provide code as well, but please elaborate your substantive answer in text for people who don't read this language well enough to recognize & extract the answer from the code. $\endgroup$ May 1, 2020 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.