1
$\begingroup$

I have many measures of Y for two groups A and B. Following the more modern approach of research in education, I calculated the effect size of the difference of Y for both groups (Cohen's d), and the significant interval for the effect size.

The fact that the effect size was positive and significant (both in statistical sense - the interval did not include the 0, and in the relevance sense - the value was large) was surprising. So I want to explain away the effect size by using a new independent variable X. The argument goes like this - the difference between A and B is due to the fact that A's have on average larger X's than B's, and it is known that larger X "causes" or is correlated to larger Y's.

If I wanted to explain away Y by using X, I would do the regression:

Y ~ C1 + C2*X + C3*AB + C4*AB*X

where AB is the categorical variable that indicates whether the group is A or B, and then show that the coefficients C3 and C4 are not significant (in the statistical sense). Thus, the coefficients that include the variable AB are not really different than 0 and thus the Y can be fully explained by the X alone.

In fact I did that, and the coefficients are significant, but I cannot interpret if they are large of not because they are in units of Y, and I want in effect size "units" (effect size do not have units and that is what makes them so useful!).

The variable X is an integer with a small range of values, so I could calculate the effect size of AB for each of the values of X, and than perform a regression on those values and X, but I find this solution odd.

In summary, given an effect size E, based on a categorical variable AB, how do I measure how much it is explained away by an independent variable X?

EDIT - after @Peter Flom's answer

I was not clear to specify that I do not know how to calculate the Cohen's d if there is a covariate variable X.

Cohen d = ($\mu(Y_a) - \mu(Y_b))/\sigma_{ab}$

where $\sigma_{ab}$ is a pooled standard deviation.

With the regression model above I now have

Y ~= C1 + C2*X + C3*AB + C3*AB*X

and I no longer have groups to calculate the mean of, or the standard variation!

I would very much like remain in the Cohen d domain. The first analysis - by calculating the d for the groups A and B was standard in the literature, and I could compare that d with others to claim that the effect was large. I would like now to show that when X is taken into account the d reduces to a very small (or hopefully not significant) number.

$\endgroup$
1
$\begingroup$

It is certainly allowable to add covariates to an equation, and to add the interaction if you like.

You can keep the effect size in units of Y, there is nothing wrong with that. If you can't interpret an effect size in Y units, then you might not understand your variables. But you can also use standardized betas (most computer packages will output it for you). I don't find these particularly useful, but some people do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.