# Cohen's d from a linear regression model

Problem:

I have a multiple regression model with categorical/binary ($c_i$) and continuous ($x_i$) variables:

$v = \alpha + \beta_1 x_1 + \beta_2 x_2 + ... + \gamma_1 c_1 + \gamma_2 c_2 +$

I am interested in the categorical binary variable $c_1$. I want to compute the Cohen D of the two groups defined by $c_1$ controlling for the other variables.

What do I know so far:

There are some formulas on how to convert F to D (for example https://www.campbellcollaboration.org/media/k2/attachments/converting_between_effect_sizes.pdf pg 13) but they assume a ANCOVA model, that is a linear model with one continuous and one categorical variable. The formula uses the r.square of the covariate (the continuous variable) in relation to the dependent variable $v$.

The formula is:

$d = \sqrt{\frac{(n1+n2)(1-r^2) F}{n1 n2}}$

This is the formula also used in the compute.es R package to convert effect sizes, but the reference they use to justify the formula Borenstein (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta analysis (pp. 279-293) also is explicit on the ANCOVA modeling (one categorical and one continuous variable.

I am not sure that the formula is valid for a multiple variable regression.

Furthermore I assume that the r.square would be of the whole regression without the categorical variable of interest ($c_1$).

Finally, the is the issue of how is the F computed. I assume that it is a Type II ANOVA, but again not 100% sure.

There was at least two similar questions on CV: Can I calculate Cohen's $d$ from multiple regression coefficient? , unanswered, and Cohen's d from regression coefficient? whose answer refer to the residual standard error which I am reasonable sure is not the correct answer to my case.

Thus my questions:

1) Does the formula holds for multiple variables regression?

2) Does the r.squared refer to the regression without the categorical variable of interest?

3) is the F calculated using a Type II ANOVA?

• What do you mean by "hold"? Is there a particular property, such as null hypothesis normality, that you want to utilize? For multiple regression, I believe that $\eta^2$ or $\omega^2$ are commonly cited statistics. I believe they are mentioned on the "Effect size" Wikipedia page. But you should know that in many fields, the use/abuse of effect sizes has been criticized, much like the use of p-values. – deasmhumnha May 27 '18 at 20:27
• I do not understand the question regarding the null hypothesis. I am not planning on performing significant tests, but to work only with effect sizes. In particular this is a research on Education and it seems to me that Cohen D is the more useful effect size, since it can be compared across experiments (for example onlinelibrary.wiley.com/doi/abs/10.1111/… ) But I would be interested and thankful on any reference on criticism of effect sizes. – Jacques Wainer May 27 '18 at 23:02
• Sorry, I misspoke. It's been a while since I was researching about effect size. It wasn't so much criticism as much as emphasis on providing confidence intervals alongside effect size point estimates as a measure of uncertainty. – deasmhumnha May 28 '18 at 0:40

In your regression model, $c_1$ (an unstandardized regression coefficient for a 0/1 indicator variable) is an adjusted mean difference, adjusting for the other variables in the model. As such, you can use $c_1$ as the numerator in the Cohen's $d$ computation, such as: $$d = \frac{B}{s_{pooled}} ~,$$ where $B$ is the coefficient $c_i$, and $s_{pooled}$ is: $$s_{pooled} = \sqrt{ \frac{ s_1^2 \left( n_1-1 \right) + s_2^2 \left( n_2-1 \right) } { n_1 + n_2 - 2 } } ~$$ The subscripts in the above are for the two groups that are part of $c_i$. The challenge is that unless you have the raw data (which you might but someone conducting a meta-analysis might not), these values are typically not reported for a complex regression model. However, $s_{pooled}^2$ is simply the overall variance for $y$, minus the variance attributed to the treatment effect (e.g., $c_i$). Thus, assuming you have both the overall standard deviation for $y$ and the sample sizes for the two groups created by the $c_i$ indicator variable, $s_{pooled$} can be computed as follows:
$$s_{pooled} = \sqrt{ \frac{s_y^2(N-1) - B^2\left(\frac{n_1n_2}{n_1+n_2}\right) } {N-2}} ~.$$
So if you aren't making an statistical arguments, what needs to "hold"? Are you asking if it accurately measures the effect of $c_1$ when controlling for other variables? Probably not, since in its standard formulation Cohen's $d$ is the mean difference normalized by an estimate of the pooled variance. On the other hand, $\eta^2$/$\omega^2$, and their partial versions, are explicitly designed to express the effect size(s) you are looking for. Here is a fairly well cited paper about a generalization that works for many common experimental design structures. This article, however, discusses ways to use pairwise t-scores to calculate Cohen's $d$ while controlling for covariates.