How do you go about calculating effect size for the Welch t-test?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ related Q, also unanswered, stats.stackexchange.com/questions/200561/… $\endgroup$– Glen_bMay 1, 2016 at 23:53
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
Depends on what is meant by "effect size" here. Welch's t-test is used to test the null hypothesis $\mu_1 = \mu_2$ when we cannot or don't want to assume that the variances are homoscedastic within the two groups.
So what is a good effect size measure to go along with this test? Obviously, it should express how different the means are in the sample. So, we could just compute the mean difference (which is an effect size measure) and there are no difficulties in computing it in this case. It's just $$y = \bar{x}_1 - \bar{x}_2,$$ whose variance can be estimated with $$\mbox{Var}[y] = \frac{SD^2_1}{n_1} + \frac{SD^2_2}{n_2}.$$
However, with "effect size", people often mean some kind of standardized measure (like Cohen's d) and the title makes it clear that this is what the asker is after. There are at least two possibilities here.
The first is to standardize the mean difference by the standard deviation from one of the two groups (e.g., if one group is a control and the other a treatment group, then we could standardize using the control group SD). So, we compute $$y = \frac{\bar{x}_1 - \bar{x}_2}{SD_1},$$ whose variance can be estimated with $$\mbox{Var}[y] = \frac{1}{n_1} + \frac{SD^2_2/SD^2_1}{n_2} + \frac{y^2}{2n_1}.$$
Alternatively, if it does not make sense to choose one of the two SDs for the standardization, then we could proceed as suggested by Bonett (2008) and standardize based on the average SD. This is computed by averaging the two variances and then taking the square-root, that is, let $$\overline{SD} = \sqrt{\frac{SD^2_1 + SD^2_2}{2}}.$$ Then we compute $$y = \frac{\bar{x}_1 - \bar{x}_2}{\overline{SD}},$$ whose variance can be estimated with $$\mbox{Var}[y] = \frac{y^2}{8(\overline{SD})^4} \left(\frac{SD_1^4}{n_1-1} + \frac{SD_2^4}{n_2-1}\right) + \frac{SD_1^2 / \overline{SD}^2}{n_1-1} + \frac{SD_2^2 / \overline{SD}^2}{n_2-1}.$$
-
$\begingroup$ heyy, I've noticed two things in your answer that made me quite curious. If I'm using only control group's standard deviation (because I think it's the best estimate I do have about population's standard deviation), does it make any sense to use both standard deviations when calculating variance? Shouldn't I just use the control group's one? $\endgroup$– ÁgathaNov 7, 2018 at 19:57
-
$\begingroup$ Also, it raises one more question: when I turn that variance into standard deviation of sample's effect size, to turn it into my margin of error I need a t score, and therefore a degree of freedom. Once I have a heteroskedastic set of samples, where I have used a Welch t-test instead of a Student's one for that reason, should I calculate the degrees of freedom based purely on n, or should I use Welch's t-test formula for degrees of freedom? $\endgroup$– ÁgathaNov 7, 2018 at 20:02
-
1$\begingroup$ I assume your first question is about the $SD^2_2/SD^2_1$ term. Yes, this makes sense (it is properly derived). As for your second question, you mean what df to use if one were to compute a confidence interval of the form $y \pm t \times \sqrt{\mbox{Var}[y]}$? Neither. I would just use $\pm 1.96$, as all of the above is based on large-sample approximations anyway. $\endgroup$– WolfgangNov 8, 2018 at 17:35
-
$\begingroup$ thanks for your answer, I didn't see it earlier. Is Bonnet (2008) the source for the 1st equation too? The only thing I'm wondering is how do I use the SD of the 2nd group for the variance, when I'm not using it for the calculation of the Glass' Delta. Somehow it looks strange (maybe because I don't have a strong background and thus I can't understand it). Anyway, thanks for your help!! $\endgroup$– ÁgathaNov 12, 2018 at 1:41
$\begingroup$
$\endgroup$
1
You don't. You calculate the effect size using the data, irrespective of the kind of T-test you used. One package in R is effsize.
d <- cohen.d (y ~ factor (x), hedges.correction = TRUE)
You can also subtract mu2 from mu1 and divide that difference by the average of the standard deviations. This will result in Cohen's d.
-
6$\begingroup$ This pools the standard deviations of the two groups, like a standard independent samples t-test; presumably OP chose to use the Welch approximation because it's not reasonable to pool the variances in this case. $\endgroup$ Jan 17, 2017 at 16:08