3
$\begingroup$

The Question

Suppose I train a predictive model on a set of features $x_1, \dots, x_n$, but for some $i \neq j$ we have $x_i = x_j$ for every data point in the training set; i.e. one of these features is a totally redundant copy of the other.

What are the consequences for learning? Does it depend on whether my model is linear or nonlinear? Does it depend on my training algorithm?

More generally, what should I expect if one of the $x_i$'s is a linear combination of the other features for every point in the training set?

My thoughts so far

Suppose the true target function is a noise-free line, $y = w_1 x_1$. Then a basic linear model will learn the parameter $w_1$ exactly. Now I duplicate the feature $x_1$ by creating a copy $x_2 = x_1$. Any combination of weights in the set $\{(\hat{w}_1, \hat{w}_2): \hat{w}_1 + \hat{w}_2 = w_1\}$ will perfectly fit the data. I'm guessing that the training algorithm will influence which particular pair is chosen.

Improve this question
$\endgroup$
3
  • $\begingroup$ Please clarify the sense of "totally redundant copy:" would this be essentially the inadvertent reproduction of a single observation in the dataset or is it perhaps an independent observation that has precisely the same values? $\endgroup$
    – whuber ♦
    Commented Nov 2, 2018 at 19:27
  • $\begingroup$ @whuber: I mean every observation has two attributes that are identical. Think copying a column in the data table. $\endgroup$
    – tddevlin
    Commented Nov 2, 2018 at 20:46
  • $\begingroup$ That's an example of collinearity. Ordinarily one would just drop one of the columns on the theory that they represent exactly the same property of the observations. (They might not, but as far as the data are concerned, they do.) $\endgroup$
    – whuber ♦
    Commented Nov 3, 2018 at 0:20

1 Answer 1

Reset to default
9
$\begingroup$

This is an instance of (multi)collinearity.

As you've noted in "my thoughts so far," ordinary linear regression can "distribute" its weight between the two columns as it pleases without changing the objective. Then fits by actually inverting a matrix may fail; if you fit by some general-purpose optimization algorithm, you'll get some arbitrary breakdown between them. This can cause serious problems if you're trying to infer regression weights, but if you just want to learn a predictive model it doesn't really matter (unless your matrix inverse fails).

If you use a regularized linear model, say ridge regression or LASSO, then the weight will be distributed between the two components in a way that corresponds to the choice of regularization.

If you're fitting a decision tree, then it'll split based on one or the other arbitrarily, and it won't really matter unless you're looking at feature importance, where it'll mess you up in the same way as weight inference in linear models.

If you're doing a method based on distances between points, e.g. nearest neighbors, kernel smoothing, or many types of Gaussian processes, then it'll effectively make that component "count more" in the distance.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ +1 but note that when you say "If you're fitting a decision tree, then it'll split based on one or the other arbitrarily, and it won't really matter," I would add that if you care about feature importances; tree ensembles will tend to divide importance between the two features, which will suppress the relative ranks of both $\endgroup$
    – Sycorax ♦
    Commented Nov 2, 2018 at 22:20
  • 1
    $\begingroup$ Good point – added to the answer. $\endgroup$
    – Danica
    Commented Nov 2, 2018 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.