1
$\begingroup$

I'm working on a project where I want to compare two groups of participants based on several different metrics. Right now, I'm estimating separate regressions with different dependent variables (i.e. the metrics I consider) but similar explanatory variables to see how they differ (e.g. a dummy for group 1, and some control variables). Is this approach okay, or should I consider something else? In all the regressions I am using fixed effects, and clustered standard errors. The sample size is not the same across all regressions, as for some metrics a particular event needs to have realized.

I'm seeking guidance on whether this methodology is appropriate or if there's a more suitable approach I should consider. Specifically, I'm curious about how to address potential correlations between the dependent variables across the two groups. Any insights or recommendations on handling this issue would be greatly appreciated. Also any papers where the researchers model the differences between groups based on different metrics (dependent variables) would be highly appreciated!

$\endgroup$
2
  • $\begingroup$ Hi, how many different regression models with same predictors you are planning to run, i.e. how many dependents do you have? $\endgroup$
    – Sointu
    Commented Apr 19 at 11:26
  • $\begingroup$ I have about 6 different regressions where the dependent variables are different, but the explanatory variables the same. Note that the nr. observations can differ across regressions though, as in order to construct some dependent variables, a certain event has to be satisfied (so it becomes a subset of the overall data). $\endgroup$
    – John
    Commented Apr 19 at 13:32

1 Answer 1

0
$\begingroup$

Right, so there are several approaches to this situation.

One is to run the 6 regressions as you planned and adjust the p-values of your group comparisons of your target variable as you usually would (with Tukey's?), otherwise do nothing. This may be fine as you are only interested in the group effect (?) and other variables are just controls.

Some would want to adjust all p-values you obtain for instance via Benjamini-Hochberg procedure.

Perhaps more sophisticated methods would include using a structural equation model in which you'd include all dependent variables at the same time, or a multilevel regression with the same idea.

Edit. so the possible issues with running 6 separate regressions are increased risk of false positives (but this is probably not a huge issue if you only look at the group comparisons and adjust p-values for them) and difficulty comparing regression coefficients across models, which you can nicely do if you run, for instance, a SEM with all dependents together. But in that case you need to have a priori idea of how all your variables are related.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.