5
$\begingroup$

I constructed a model via multinominal logistic regression analysis. The final model contains three predictors. All predictors are significant when they are the only predictors. However, the coefficient of one of the predictors is not significant when included all three predictors are included in the model.

Should I include this predictor in the final multinomial logistic regression equation?

$\endgroup$
2
  • $\begingroup$ have you checked the standard diagnostics for collinearity? $\endgroup$
    – Macro
    Commented Sep 28, 2011 at 2:32
  • $\begingroup$ This is almost the same as the question at stats.stackexchange.com/q/14500/919 which concerns the same phenomenon in multivariate regression generally. The answers there apply here, too. $\endgroup$
    – whuber
    Commented Sep 28, 2011 at 21:09

3 Answers 3

8
$\begingroup$

It depends whether you are doing... a) predictive research, where you don't care about what is causally responsible, only what serves as an efficient set of indicators, or b) explanatory research, where you want to disentangle causal relationships as much as you can.

In the latter, when multiple correlated predictors vie for a role in your equation, you would care about such things as giving "causal credit" to earlier factors over later ones, since what comes later could never cause what came before, but sometimes the reverse is true. You would care about giving more "credit" to relatively objective, relatively fixed variables such as marital status or ethnicity than to relatively subjective, changeable ones such as attitudes and opinions. And (and here I'm paraphrasing James Davis's The Logic of Causal Order) you would want to choose more generative factors such as socioeconomic status over less generative ones such as what brand of toothpaste a person uses.

When your candidate predictors are correlated, no statistical algorithm (such as a stepwise regression) can deal with these issues of explanation. It is up to you as a researcher to think through your candidate variables and choose those that will best serve your purpose. It is only in pure predictive research that you can ignore such issues and simply choose those predictors that account for the most variance in the outcome--or, in your case, produce the highest pseudo-r-squared.

Your question gets to the heart of important issues in multivariate modelling of many types, and if more than 5 tags were allowed I would have also listed multicollinearity, model-building, and/or variable selection.

$\endgroup$
2
  • $\begingroup$ +1 I'd always wondered what the right/least-wrong way to do this is. $\endgroup$ Commented Sep 28, 2011 at 4:09
  • 1
    $\begingroup$ I don't think that maximizing pseudo-r-squared is a good strategy, although i agree that model goodness-of-fitting is important to assess the model... $\endgroup$ Commented Sep 28, 2011 at 12:11
2
$\begingroup$

As @rolando2 mentioned, this depends very much on what your trying to accomplish or what question(s) you are trying to answer.

If you are trying to find a good model for prediction then rather than just deciding on whethere to include a term or not, it is better to use some type of shrinkage method such as penalized regression, ridge regression, lasso/lars, or model averaging.

You should also take into account outside knowledge about the variables. If my doctor had a choice of 2 predictive models to help in diagnosing me I would prefer that he use the one that uses blood pressure as a predictor rather than the one that uses the results from an exploratory surgery, even if it has a slightly smaller $R^2$ value.

$\endgroup$
1
  • $\begingroup$ Surgery vs. blood pressure: nice example. $\endgroup$
    – rolando2
    Commented Sep 28, 2011 at 21:14
2
$\begingroup$

If predictive accuracy is the main objective, then it is generally better to use regularisation to address problems such as correlated predictor variables and not perform any feature selection. This is because feature selection is difficult. Most often feature selection is peformed by optimising some feature selection criterion evaluated over a finite dataset. Since only a finite dataset is used, the feature selection criterion has a non-zero variance, and hence it is possible to over-fit the feature selection criterion (and get a set of features that is optimal for this particular sample of data, but not for the true underlying distribution and hence generalisation is poor). Over-fitting is always most dangerous when you have many degrees of freedome with which to optimise the criterion, and in feature selection, there is one per feature. For regularisation (e.g. ridge regression or regularised logistic regression) there is only one degree of freedom (the ridge parameter) and so the risk of over-fitting is generally lower (but it doesn't go away completely). This is the advice given in the appendix of Millar's monograph "subset selection in regression" (but without the reasoning IIRC).

If you can identify the variables that are the causal "parents" of the quantity you seek to predict, then using only those featrues has the advantage that the model will still work well when extrapolating or under covariate shift (e.g. the sampling of the data uses a different distribution), as your model will represent the true causal structure, rather than mere correllations. So if extrapolation or covariate shift is an issue, causal feature selection may be helpful (although in practice identifying causal relationships is unreliable). Isabelle Guyon has much to say that is well worth listening to on this topic (just found a videolecture here that I am going to watch now).

There is no need for the same model to be used for explication and for prediction, so I would say fit two models, one with feature selection to help you understand the problem/data and a second model with no feature selection but with properly tuned regularisation to use for prediction.

$\endgroup$
2
  • 1
    $\begingroup$ Regularization methods can not generally be seen as having one degree of freedom. The relevant concept is that of effective degrees of freedom, which sometimes can be defined, e.g. for ridge regression, and which is generally an monotonely decreasing function of the penalization parameter. $\endgroup$
    – NRH
    Commented Sep 28, 2011 at 18:30
  • $\begingroup$ True, it would be more correct to say something along the lines that regularisation methods constrain the hypothesis class more strongly than feature selection, however the paragraph was getting a bit unwieldy already (but point taken). The essential point is that feature selection is generally more prone to over-fitting the selection criterion than regularisation methods. $\endgroup$ Commented Sep 28, 2011 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.