3
$\begingroup$

When do I use Cohen's and when do I t-test? Probably in addition: What is the (conceptual) difference between them? Both tests are meant to study the difference between two distributions. I just roughly know that Cohen's is used to calculate the effect size while t-test is meant to study whether there is a general difference between two distributions(?).

Formulas:

t-test: $ T = \frac{\bar{x} - \bar{y}}{ \sqrt{\frac{s_x^2}{n_x} + \frac{s_y^2}{n_y}} } $

with $ s $ as a variance.

Cohen's $d = \frac{\mu_1 - \mu_2}{ \sigma } $

I see there are differences but it also looks very similar. Isn't it?

img

Improve this question
$\endgroup$
5
$\begingroup$

Cohen's d seeks to tell you how big the standardized difference is between the two distributions. It's very popular in areas like psychology where I think there are no obvious units you can use to describe the difference. In medical stats, I could say (for example) that your HbA1c levels were on average 5mg different in the two groups, and wouldn't need to use Cohen's d.

The t-test is an attempt to tell you have enough evidence to reject the idea that the difference is non-zero. However, a non-zero difference could be, in practical terms, completely irrelevant. Also, don't forget you have to make technical assumptions when using the t-test, e.g. you default to assume the two groups have the same variance.

There are arguments that it is more useful to compare confidence or credible intervals estimated from the two samples.

There's an interesting article here: https://bmcresnotes.biomedcentral.com/articles/10.1186/s13104-015-1020-4

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks, so Cohen's d is basically Cohen's/standard deviation? I will have a look at the link! $\endgroup$ – Ben Nov 20 '19 at 10:47
  • $\begingroup$ Yes. Difference in means divided by standard deviation. If there were units in the data, it would be unit-less. $\endgroup$ – Paul Hewson Nov 20 '19 at 10:49
  • $\begingroup$ it's somehow similar to covariance and pearson correlation.. it is also so close to each other that I'm always having a hard time figuring out why such concepts are handled as if they were something on their own. $\endgroup$ – Ben Nov 20 '19 at 10:58
  • 1
    $\begingroup$ That's a profound comment. Yes, correlation is standardised covariance. I blame outdated text books, the GAISE proposals in the US are much more focussed on concepts. $\endgroup$ – Paul Hewson Nov 20 '19 at 11:01
  • 1
    $\begingroup$ "So Cohen's d is basically Cohen's/standard deviation?" No, that wouldn't make sense. Cohen's d is {the mean difference between groups} divided by standard deviation (either the pooled version or the control- or comparison-group's standard deviation). $\endgroup$ – rolando2 Nov 20 '19 at 12:36
1
$\begingroup$

T-test is in complimentary relation with Cohen's $d$ (and equivalence tests using Cohen's $d$).

T-test gives a p-value which is the probability of committing a Type I error. One can reject the null hypothesis, if the p-value is too small, but one cannot claim that the null hypothesis is true on the basis of p-value only, without risking of making a Type II error.

To assess the risk of Type II error one has to perform power testing, i.e. calculating the probability that the alternative hypothesis is correct. However, in the case of testing $H_0 : \mu =0$ against $H_1 : \mu \neq 0$ direct power calculation is impossible.

One common solution to this problem is assuming the minimal size of the effect (here is where Cohen's $d$ comes in) and proving that the actual effect is smaller than this minimal size. This is known as "equivalence testing" or TOST (two one-sided tests). Here is a useful reference: https://www.ncbi.nlm.nih.gov/pubmed/28736600

There are alternative approaches, e.g., based on the use of Bayes factors. But these would take us too far from the core of your question.

Improve this answer
$\endgroup$
5
  • $\begingroup$ from an application point of view: I should use them always together? The one for the significance and the other one for the effect size? $\endgroup$ – Ben Nov 21 '19 at 8:47
  • 1
    $\begingroup$ Not necessarily. If, e.g., you are trying only to prove that there is an effect (i.e. the two groups have different means), then the small p-value is enough. Proving that there is no effect or dealing with p-value that is not small is when you need to do both. $\endgroup$ – Vadim Nov 21 '19 at 8:58
  • $\begingroup$ But when I want to know how strong the effect is, I need Cohen? Afaik with the size of a dataset the possibility that there is an (significant) effect, is increasing, right? $\endgroup$ – Ben Nov 21 '19 at 10:02
  • 1
    $\begingroup$ Given a big dataset, even a very small effect can become significant. So figuring out what size of effect is meaningful is always a good idea. Btw, I strongly recommend the article that I cited in my answer. $\endgroup$ – Vadim Nov 21 '19 at 10:20
  • $\begingroup$ Right, sorry, I missed that linked article.. will have a look at it. $\endgroup$ – Ben Nov 21 '19 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.