2
$\begingroup$

I am creating a sports prediction model with around 300 features. There is a high degree of multicollinearity which breaks an assumption for logistic regression. The issue is that several of these features contain the most information toward prediction. One such predictor is the current number of wins a team has, which can be indirectly predicted by the other predictors. Can I keep this feature, and several other important features with high multicollinearity?

$\endgroup$
3
  • 4
    $\begingroup$ Why do you think there is such an assumption? $\endgroup$
    – Michael M
    Commented Sep 12, 2023 at 12:32
  • $\begingroup$ @MichaelM My understanding was that in order for parameter coefficients to be accurate, and hence deliver accurate predictions, several assumptions must be met. Namely, linearity of the logit and the continuous features, absence of multicollinearity, and no overly influential outliers. Why does this assumption not need to hold for a prediction problem? $\endgroup$
    – sla813
    Commented Sep 13, 2023 at 0:21
  • 3
    $\begingroup$ Multicollinearity blows up the variance of the coefficients (and makes Ceteris Paribus interpretation of them unnatural). But the predictions themself (and goodness of fit measures) are not affected. Since you are mainly interested in a prediction machine, this multicollinearity should not be an issue (except from introducing certain unnecessary overfitting). Absence of multicollinearity is not an assumption. Otherwise, one would probably not use logistic regression: predictors are always at least slightly correlated. $\endgroup$
    – Michael M
    Commented Sep 13, 2023 at 4:57

3 Answers 3

9
$\begingroup$

There is no assumption about uncorrelated features in logistic regression. For instance, people will run logistic regression on the MNIST digit pixels without giving it a second thought and achieve decent performance north of $90\%$ accuracy. While this is hardly world-class performance (deep learning approaches can score more like $99.97\%$ accuracy), that’s a lot of digits identified correctly. However, some of the pixels are quite correlated.

I understand the appeal of dropping some correlated predictors: preserve most of the “information” available in the features yet drop a parameter that risks overfitting. However, there are a few drawbacks to this.

  1. Unless there is perfect multicollinearity, the dropped feature will contain unique information. Yes, dropping a parameter can help to quell overfitting, but that can come at the expense of underfitting! Imagine dropping a feature due to overfitting concerns and wanting strong predictive ability, only to find yourself unable to get a good fit because a useful feature has been dropped from the model.

  2. It is possible to overfit a feature selection step. If we consider overfitting to be fitting to coincidences in the data that do not generalize, a selection of particular features can be specific to a particular set of training data, and you might get a rather different selection of features in a different split of the data. Our Frank Harrell talks about the instability of feature selection in one of his machine learning presentations.

  3. If you start including interaction terms, which are implicit in many sophisticated machine learning models like neural networks and random forests and can be included in logistic regression models when those interactions are made explicit, dropping a variable misses the opportunity to have it participate in the interactions.

  4. There are techniques, such as regularization, that can reduce overfitting yet do not totally remove the variable from the analysis.

Therefore, it is far from a given that you should remove a feature that is correlated with other features.

$\endgroup$
5
$\begingroup$

I don't know of any problem with collinearity if your only goal is prediction. But make sure that is your only goal. The parameter estimates can be totally misleading.

One negative feature of collinearity is that small variation in the data can yield huge variation in the model. Belsley (in one of his books, I forget which) gives an example where changing data in the third significant figure flips the parameter estimates from positive (and sig) to negative (and sig). But the models make nearly identical predictions.

The danger is that you start thinking about your model as if the parameter estimates are, somehow, good. "Oh! Teams that do XXX win more games" but that may not be the case.

With 300 features, overfitting may also be a problem, but that's separate from collinearity and you may have a huge database of games.

$\endgroup$
1
  • 1
    $\begingroup$ Of course, even if there's no doubt about a coefficient's sign, there's the danger of interpreting "teams that do XXX win more games" as "If a team did XXX vs. not, they would win more games", when it might just be that teams that do XXX also happen to be the ones that do XXX. E.g. people that put out rat poison on their property, likely have more rats on their property 1 week later than those that do not. However, that's not because rat poison makes rats live longer or causes more rats to turn up or causes them to have more baby rats, but rather because all else was not equal to start with. $\endgroup$
    – Björn
    Commented Sep 13, 2023 at 11:58
3
$\begingroup$

A lot of literature regarding multi-collinearity being problematic refer to either

a) difficulty in interpreting coefficients (arguably of less importance in a model that's primarily for prediction),

b) difficulty in finding a maximum likelihood estimate for model parameters in a traditional statistical model (e.g. complete or quasi-complete separation in logistic regression - which is a problem without regularization, but some other model types deal with it more "gracefully" e.g. random forest, gradient boosted decision trees),

c) inefficiency (the more highly similar multiple features are, the more you wonder whether there's a lower dimensional representation that captures almost the same information and then some issues go away, such as in gradient boosted decision trees the regularizing effect of sub-sampling columns is reduced if you have many almost identical columns), or

d) overfitting (too many features and too little training data = potential to have some randomly associated with the target variable by chance = overfitting risk, if not dealt with via a robust process).

One can debate how much any of these truly matter for a model that is solely for prediction (e.g. explainability/interpretability may still be rather important).

However, beware some proposed solutions that are known to be problematic (e.g. pre-screening each feature vs. the target in isolation aka "univariate feature selection").

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.