1
$\begingroup$

I am investigating whether the predictive validity of the Big Five personality traits can be improved by controlling for sources of self-report error (e.g. memory, interpretation biases, ect.). I am specifically interested in identifying the confounding variables with the strongest effects.

To ensure that I am testing general differences in predictive validity, I would like to consider three outcome variables simultaneously (a,b,c), known to correlate strongly with two of the Big Five traits (Neuroticism and Extraversion)

Thus, I conducted several hierarchical multivariate regressions in the following form:

BaseModel <- lm(cbind(a, b, c) ~ Neuroticism + Extraversion, data)

Confound.1.Model <- cbind(a, b, c) ~ Neuroticism + Extraversion + Confound1, data)

anova(BaseModel,Confound.1.Model)

Confound.2.Model <- cbind(a, b, c) ~ Neuroticism + Extraversion + Confound2, data)

anova(BaseModel,Confound.1.Model)

Now, both of the ANOVAs tell me that the confounds explain incremental variance in the set of outcomes, over and above personality. However, I do not know how much more variance is explained -- that is to say, the R-Square change metric is not provided, as it normally would be in a multiple regression with one outcome variable.

Is there some valid way to calculate the R-square change for a multivariate regression? (e.g. averaging the R-square changes for the individual outcomes).

Is there some other statistic that is better suited for my question?

Thank you for your time.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Without knowing the nature of the dependent variables, I would initially suggest the following.
   First, even if there is a correlation between the errors for the set of dependent variables, it may not be informative to assess confounding in a multivariate setting. In other words, you can answer the question of confounding (or degree of confounding) with multiple univariate analyses.
   Second, if there is a substantive reason to examine the correlated errors across the set of dependent variables (e.g., different temporal measurements of the same construct), then I would recommend using structural equation modeling in which the dependent variables are predicted by a latent variable, and the independent variables serve as predictors for the latent variable. (Note, this almost is a MIMIC model, but in truth, it actually is structurally equivalent to a CFA...happy to speak more to that in comments if relevant.) In this case, you can obtain an $R^2$ for the latent variable, and you can do an F-test with the two different models (2 iv's vs. 3 iv's).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Unfortunately, I cannot run a CFA because the three DVs that I measured represent vastly different concepts that are correlated with Extraversion and Neuroticism (e.g. Health and Emotional Stroop Task performance). It looks like I have to stick with the HR. Thanks! $\endgroup$
    – unicoder
    Mar 18, 2018 at 16:00
  • $\begingroup$ If they are not theoretically comparable constructs, then this is reasonable. If you want to be judicious in evaluating whether or not the observed confounding is truly a consequence of the confounding variables or some attribute of shared variance across the errors of the dependent variables, (1) you could run separate univariate analyses, (2) examine the $∆ R^2$ for each, (3) run a PCA with the dep. var.s, (4) run the analysis with this as the dep. var., (5) calculate this $∆ R^2$. This is not a rigorous assessment, but it may suggest whether a difference is worth exploring. $\endgroup$
    – Gregg H
    Mar 18, 2018 at 19:35
0
$\begingroup$

What I normally see in papers using hierarchical multiple regression is simply the arithmetic difference between the $R^2$s between the models of interest.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! Could you please provide a citation to one of these papers? $\endgroup$
    – unicoder
    Mar 18, 2018 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.