14
$\begingroup$

I was advising a research student with a particular problem, and I was keen to get the input of others on this site.

Context:

The researcher had three types of predictor variables. Each type contained a different number of predictor variables. Each predictor was a continuous variable:

  • Social: S1, S2, S3, S4 (i.e., four predictors)
  • Cognitive: C1, C2 (i.e., two predictors)
  • Behavioural: B1, B2, B3 (i.e., three predictors)

The outcome variable was also continuous. The sample included around 60 participants.

The researcher wanted to comment about which type of predictors were more important in explaining the outcome variable. This was related to broader theoretical concerns about the relative importance of these types of predictors.

Questions

  • What is a good way to assess the relative importance of one set of predictors relative to another set?
  • What is a good strategy for dealing with the fact that there are different numbers of predictors in each set?
  • What caveats in interpretation might you suggest?

Any references to examples or discussion of techniques would also be most welcome.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
13
$\begingroup$

Suppose that the first set of predictors requires $a$ degrees of freedom ($a \geq 4$ allowing for nonlinear terms), the second set requires $b$, and the third requires $c$ ($c \geq 3$) allowing for nonlinear terms. Compute the likelihood ratio $\chi^2$ test for the combined partial effects of each set, yielding $L_{1}, L_{2}, L_{3}$. The expected value of a $\chi^2$ random variable with $d$ degrees of freedom is $d$, so subtract $d$ to level the playing field. I.e., compute $L_{1}-a, L_{2}-b, L_{3}-c$. If using F-tests, multiple F by its numerator d.f. to get the $\chi^2$ scale.

Improve this answer
$\endgroup$
6
  • $\begingroup$ To confirm, your approach is to compute L1 as the reduction in deviance (-2*) resulting from the inclusion of the four social variables, adjusted by the df of these four variables? And likewise in turn for L2 and L3? $\endgroup$
    – B_Miner
    Commented Apr 27, 2011 at 12:57
  • $\begingroup$ I didn't use the best notation. I mean the likelihood ratio $\chi^2$ statistic, which is the change in -2 log likelihood upon removing the set of variables being tested. $\endgroup$
    – Frank Harrell
    Commented Apr 28, 2011 at 1:54
  • $\begingroup$ would you also grant that there's a risk, in devising a purely statistical solution, of missing a possible overarching problem whereby all 3 groups of predictors could be measuring characteristics/behaviours occurring at the same time. Without an earlier-causes-later sort of basis for a causal chain, might it be impossible to definitively disentangle causal relationships in this situation--whatever our calculations might be? (I'm trying to think the way James Davis does in The Logic of Causal Order.) $\endgroup$
    – rolando2
    Commented Apr 30, 2011 at 1:05
  • $\begingroup$ For sure. The causal chain has to be understood before modeling even commences. $\endgroup$
    – Frank Harrell
    Commented May 14, 2011 at 16:02
  • $\begingroup$ @FrankHarrell Do these results apply to the penalized likelihood as well? Does the penalized likelihood have any properties that make it differ from the likelihood with respect to this variable-importance measure? Could you suggest any papers that describe this in greater detail? Thanks. $\endgroup$
    – julieth
    Commented Jul 13, 2012 at 19:33
12
$\begingroup$

Suggestions

  • You could perform individual multiple regressions for each type of predictor, and compare across multiple regressions, adjusted r-square, generalised r-square, or some other parsimony adjusted measure of variance explained.
  • You could alternatively explore the general literature on variable importance (see here for a discussion with links). This would encourage a focus on the importance of individual predictors.
  • In some situations hierarchical regression may provide a useful framework. You would enter one type of variable in one block (e.g., cognitive variables), and in the second block another type (e.g., social variables). This would help answer the question of whether one type of variable predicts over and above another type.
  • As a side examination, you could run a factor analysis on the predictor variables to examine whether the correlations between predictor variables map on to the assignment of variables to types.

Caveats

  • Types of variables such as cognitive, social, and behavioural are broad classes of variables. A given study will always include only a subset of the possible variables, and typically such a subset is small relative to the possible variables. Furthermore, the measured variables may not be the most reliable or valid means of measuring the intended construct. Thus, you need to be careful when drawing the broader inference about the relative importance of a given type of variable over and beyond what was actually measured.
  • You also need to consider any bias in the way that the dependent variable was measured. Particularly in psychological studies, there is a tendency for self-report measures to correlate well with self-report, ability with ability, other-report with other report, and so on. The issue is that the mode of measurement has a large effect over and beyond the actual construct of interest. Thus, if the dependent variable is measured in a particular way (e.g., self-report), then don't over-interpret larger correlations with one type of predictor if that type also uses self-report.
Improve this answer
$\endgroup$
1
  • $\begingroup$ I enjoyed reading this clear, helpful response and am going to share it with a colleague. $\endgroup$
    – rolando2
    Commented Apr 30, 2011 at 0:54
10
$\begingroup$

Importance

First thing to do is operationalise 'importance of predictors'. I shall assume that it means something like 'sensitivity of mean outcome to changes in predictor values'. Since your predictors are grouped then sensitivity of the mean outcome to groups of predictors is more interesting than a variable by variable analysis. I leave it open whether sensitivity is understood causally. That issue is picked up later.

Three version of importance

Lots of variance explained: I'm guessing that psychologists' first port of call is probably a variance decomposition leading to a measure of how much outcome variance is explained by the variance-covarance structure in each group of predictors. Not being an experimentalist I can't suggest much here, except to note that the whole 'variance explained' concept is a bit ungrounded for my taste, even without the 'which sum of which squares' issue. Others are welcome to disagree and develop it further.

Large standardized coefficients: SPSS offers the (misnamed) beta to measure impact in a way that is comparable across variable. There are several reasons not to use this, discussed in Fox's regression textbook, here, and elsewhere. All apply here. It also ignores group structure.

On the other hand, I imagine that one could standardise predictors in groups and use covariance information to judge the effect of a one standard deviation movement in all of them. Personally the motto: "if a things not worth doing, it's not worth doing well" damps my interest in doing so.

Large marginal effects: The other approach is to stay on the scale of the measurements and calculate marginal effects between carefully chosen sample points. Because you are interested in groups it is useful to choose points to vary groups of variables rather than single ones, e.g. manipulating both cognitive variables at once. (Lots of opportunity for cool plots here). Basic paper here. The effects package in R will do this nicely.

There are two caveats here:

  1. If you do that you will want to watch out that you are not choosing two cognitive variables that while individually plausible, e.g. medians, are jointly far from any subject observation.

  2. Some variables are not even theoretically manipulable, so the interpretation of marginal effects as causal is more delicate, though still useful.

Different numbers of predictors

Issues arise due to the grouped variables covariance structure, which we normally try not to worry about but for this task should.

In particular when calculating marginal effects (or standardized coefficients for that matter) on groups rather than single variables the curse of dimensionality will for larger groups make it easier for comparisons to stray into regions where there are no cases. More predictors in a group lead to a more sparsely populated space, so any importance measure will depend more on model assumptions and less on observations (but will not tell you that...) But these are the same issues as in the model fitting phase really. Certainly the same ones as would arise in a model-based causal impact assessment.

Improve this answer
$\endgroup$
2
$\begingroup$

One method is to combine the sets of variables into sheaf variables. This methods has been used extensively in sociology and related areas.

Refs:

Whitt, Hugh P. 1986. "The Sheaf Coefficient: A Simplified and Expanded Approach." Social Science Research 15:174-189.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.