Suppose that we want to fit a model to predict a given response variable $Y$. Suppose that some explanatory variables are redundant. Be an explanatory variable redundant if it gives similar information of another available explanatory variable. For example if $x_1$ is a count variable, a redundant variable would be the indicator function: $x_2=1(x_1>=1)$. If $x_3$ is a non negative quantitative variable, a redundant variable would be the indicator function: $x_4=1(x_3>0)$.
My questions are:

  1. should redundant explanatory variables be discarded from the training set? If so, why can't we add all the explanatory variables and let the variable selection algorithms (for instance, forward stepwise for linear models, MARS building procedure, lasso and CART's variable selection ability) choose which variables are worth being inside the model and which are not?
  2. if redundant explanatory variables should be removed, what is the correct way to proceed? Should I add only $x_1$ and $x_3$ (without $x_2$ and $x_4)$, see the results, then add only $x_2$ and $x_4$ (without $x_1$ and $x_3)$, see the results and finally decide which predictors to use?

Intuitively, I don't see how the prediction error can decrease by adding more explanatory variables. In other words, I don't understand why we shouldn't consider redundant explanatory variables too. As far as inference is concerned, if redundant predictors aren't discarded from the training set, can I get contradictory results (for example, in a linear model, a positive regression coefficient for $x_1$ but a negative coefficient for for $x_2$? If so, is this the only reason why redundant predictors should be removed?


3 Answers 3


Unless predictors are almost completely redundant, better predictive performance results when the competing predictors are combined, as compared to deleting predictors up front. When redundancy is very high, deletion up front can be a good idea. the R Hmisc package redun function is one approach, measuring redundancy using a flexible additive nonlinear model to predict each variable from all the (remaining) variables.

Keep in mind that assessment of redundancy must be done using only unsupervised learning techniques.


The issue you describe with a linear model where coefficients for correlated features can cancel each other out resulting in large positive/negative pairs is known as multicollinearity and can be a problem, especially if the correlation structure shifts over time. I say correlation here since it requires a linear relationship between the variables.

Related issues can arise with non linear relationships between features. Often the amount of noise or extraneous information in the features varies (for example height is a noisy proxy for age in children) so the best model will use only the features that contain the information relevant to the target with minimal noise (ie it's better to predict "can drive" from age then from height.)

The various algorithms you describe are meant to and can deal with this to some extent.

Random forests are particularly good at it as they internally repeat the simple CART feature selection algorithm over bootstrapped copies of the data set. They are often the easiest thing to get to work on highly dimensional data where this sort of issue is common/likely.

Doing an explicit dimensionality reduction step like PCA or some sort of nonlinear manifold learning is also common.

There are also a number of methods for doing explicit feature selection as a preprocessing step including things like the iterative process of adding and removing features to see how it changes performance.

With any of these methods care must be taken to avoid overfitting. The feature selection must be considered part of the overall model generating process and done and tuned within validation. Either a separate holdout or nested cross validation system must be used or the chances of ending up with model performance that does not generalize to new data is high (and goes up with the number of features in the data set).


From a practical perspective, I have made the experience that most of the methods you mention (lasso, CART, MARS, etc.) remove redundant predictors to some extent.

However, even applying those methods, redundant predictors can still have a negative impact on your out-of-sample predictive performance. Besides overfitting, usually throwing in all kinds of predictors you find increases to noise-to-signal ratio. And this makes it harder for all kinds to methods to create a good predictive model - even for the powerful methods like random forests.

So it is important to reflect on the model you are building. One the other hand, the model-building human will not always know which variables are reasonable predictors.

There is no free lunch here, unfortunately, and the best is usually to experiment a lot and learn to know your data well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.