0
$\begingroup$

This question is mainly theoretical, but based on problems I have experienced when previously designing research.

I want to know whether there is an approved statistical method to compare the effect sizes of two between-groups differences, involving one categorical independent variable and two different continuous dependent variables. In other words, does the independent variable predict larger between-groups differences in dependent 1 or dependent 2? For example:

1) The effect of gender (male versus female) on mean height (dependent 1) and mean happiness (dependent 2).

2) Between-groups differences in people responding "Yes" versus "No" on the question "Are you depressed?" in their scores on a depression questionnaire (dependent 1) and anxiety questionnaire (dependent 2).

I want to know if there are statistics to hypothesis test whether the independent variable has a greater effect on dependent variable 1, dependent variable 2, or whether it has an equivalent effect. Using example 1 above: Does gender predict height more than it does happiness?

This presumably involves effect size, but I'm not aware of any specific techniques. Mixed 2x2 ANOVA requires the two dependent variables to use the same scale of measurement (repeated measures), so that would not work.

$\endgroup$
1
$\begingroup$

There's a few different approaches you could take here. If you are worried that a mixed 2 x 2 ANOVA wouldn't work because they have to be on the same scale of measurement, you could always z-score the two dependent variables so they would be on the same mean = 0 and standard deviation = 1 scale. From there, I might also consider doing a multilevel model, looking at the interaction between, for example, gender (male or female) and outcome (happiness or height).

But it seems like you want to know the differences between the absolute value in effect, right? Let's say happiness is affected at a Cohen's d of -0.5, while height is affected at a d of 0.5. Those would be equivalent, since neither of the effects are greater, in a sense of being further away from a null effect.

I like Cohen's d as a metric, so I'll stick with a simpler approach than a multilevel model here. I would bootstrap the difference between the two Cohen's ds. I did a small simulation to demonstrate this.

First, I simulated happiness and height data for men and women. Men have height mean 69 inches, height standard deviation of 3, happiness mean of 4 and standard deviation of 1; women have mean height of 64 inches, mean happiness of 4.4, and the standard deviations here are the same as they are for men. I'm using R to do this simulation:

# simulate data ----------------------------------------------------------------
set.seed(1839)
height_m <- rnorm(100, 69, 3)
happy_m <- rnorm(100, 4, 1)
height_w <- rnorm(100, 64, 3)
happy_w <- rnorm(100, 4.4, 1)
dat <- data.frame(
  gender = factor(c(rep("m", 100), rep("f", 100))),
  happy = c(happy_m, happy_w),
  height = c(height_m, height_w)
)

Now, we can look at the Cohen's d for each:

# cohen's d --------------------------------------------------------------------
library(effsize)
height_d <- unname(cohen.d(height ~ gender, dat)$est)
happy_d <- unname(cohen.d(happy ~ gender, dat)$est)
height_d
happy_d

height_d returns 1.46, meaning that men are 1.46 standard deviations taller than women. happy_d returns -0.31, meaning that men are 0.31 standard deviations less happy than women. Since we want to know which has the greater effect, we could get the absolute value of these and then subtract happiness from height:

abs(height_d) - abs(happy_d)

Which gives us 1.15. This means that the effect of gender on height is 1.15 standard deviations larger than the effect of gender on happiness (where the standard deviations are in the original metric of the dependent variable). You could also maybe call this 1.15 Cohen's d units or something.

How do we know if this is significantly different from zero? One approach is to bootstrap a confidence interval around this difference. This code below can do this:

# bootstrap absolute value difference ------------------------------------------
set.seed(1839)
bs_diffs <- sapply(1:5000, function(placeholder) {
  cases <- sample(1:nrow(dat), nrow(dat), TRUE)
  tmp <- dat[cases, ]
  abs(unname(cohen.d(height ~ gender, tmp)$est)) -
    abs(unname(cohen.d(happy ~ gender, tmp)$est))
})

Then all one has to do is get the mean and 95% confidence intervals from this bs_diffs return:

# estimate and confidence interval ---------------------------------------------
mean(bs_diffs)
mean(bs_diffs) - 1.96 * sd(bs_diffs)
mean(bs_diffs) + 1.96 * sd(bs_diffs)

Those three lines will give us the estimate of the difference, the lower bound, and the upper bound. They return a Cohen's d difference = 1.15 and 95% CI [.73, 1.58], showing that gender affects height more than it does happiness.

$\endgroup$
  • $\begingroup$ Sorry Mark, I previously lacked the privilege to respond. Your answer is terrific. Thank you. $\endgroup$ – Josh Blake Jan 25 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.