Using a t-test to test effect size

Question

In my line of work, I work with large data and often run stat tests to compare differences between groups. The problem I am facing is that if I use a $t$-test to measure any difference, the result will almost always be significant due to the large sample sizes.

For example, I am looking to measure the effect size of a test group against a control group. Both of my sample sizes are very large - my test has $100,000$ people and my control has $11,000$ people. My $t$-test value is over $1$ million.

To account for this, I've come up with the idea of comparing the mean difference of the groups against a benchmark value. The benchmark value can be say $10$% of the mean value of the control group. So, I compare the mean difference between the test vs control groups, against the benchmark value. This way I am testing to see if the difference between the groups is greater than $10$% of the control group and whether it is statistically significant.

Is this a valid approach?

Answer 1

I'm not understanding why you don't use Cohen's $d$ or Hedges $g$ in this situation. It is specifically designed for this purpose...to measure the magnitude of the effect rather than testing the null hypothesis with probability (here with $t$ values). If you are specifically concerned about some benchmark values to establish whether the effect is "large enough", Cohen came up with some rough criterion for weak ($d = .2$), moderate ($d = .5$), and large values ($d = .8)$ a long time ago, though that can be field-specific and may require knowing more about what effects are more typical in your area of research.

This graphic here does a good job of showing how Cohen's $d$ measures the magnitude of differences if that helps in any way.

Answer 2

If you get a very small p-value, it means that your sample size was probably adequate to detect the real effect size in the population. So you can already calculate an estimate of this effect size in your sample (whatever effect size you choose: simple difference between the means of the two groups, Cohen's d, or other effect sizes); you could also compute a confidence interval around this effect size. No need for further hypothesis testing here.

That being said, you may want to compare the effect size observed in your sample to an effect size you find interesting enough, as your question suggests.

You ask if your approach is valid. A problem with thinking in terms of percentages as you do, rather than in terms of absolute values, is that it can be very difficult to judge if the difference between the test and control groups is actually important or not. Ultimately, it may depend on the actual absolute values of your control and test groups.

For instance, imagine we'd use your 10% criteria in a study about weight loss. It means that if the weight loss is 1 kg in your control group and 1.1 kg in the intervention group (a 10% difference), you would find it as interesting as a difference between 15 kg and 16.5 kg (10% difference too). Would you really consider a 100 grams (0.22 lb.) difference as interesting as a 1.5 kg (3.3 lb.) difference? I'd strongly doubt that.

In addition, how would you determine an effect size of interest if the weight loss is 0 in the control group? 10% of 0 is still 0.

So you see that at some point, we probably need to relate your 10% criteria to some absolute values to make a judgment call.

The gist of the issue is to define what is an interesting effect size in the context of your study, in absolute terms. It's not really a statistic question, but rather a domain knowledge question.

Where I'd disagree a bit with Shawn's answer is that I think you should stay away from benchmarks designed by other people for other contexts (even Jacob Cohen paradoxically advised against using the "small/medium/large" benchmark he designed; for more info, see this discussion or this discussion).

Thinking in terms of costs and benefits can help relative to that, e.g. how much real-life benefits would we get by generalizing the intervention in the population, given the effect size we observed? Does the magnitude of the effect size justify the costs of the intervention? etc.