# What happens if I train a model on a data set that includes a duplicated feature?

The Question

Suppose I train a predictive model on a set of features $$x_1, \dots, x_n$$, but for some $$i \neq j$$ we have $$x_i = x_j$$ for every data point in the training set; i.e. one of these features is a totally redundant copy of the other.

What are the consequences for learning? Does it depend on whether my model is linear or nonlinear? Does it depend on my training algorithm?

More generally, what should I expect if one of the $$x_i$$'s is a linear combination of the other features for every point in the training set?

My thoughts so far

Suppose the true target function is a noise-free line, $$y = w_1 x_1$$. Then a basic linear model will learn the parameter $$w_1$$ exactly. Now I duplicate the feature $$x_1$$ by creating a copy $$x_2 = x_1$$. Any combination of weights in the set $$\{(\hat{w}_1, \hat{w}_2): \hat{w}_1 + \hat{w}_2 = w_1\}$$ will perfectly fit the data. I'm guessing that the training algorithm will influence which particular pair is chosen.

• Please clarify the sense of "totally redundant copy:" would this be essentially the inadvertent reproduction of a single observation in the dataset or is it perhaps an independent observation that has precisely the same values? – whuber Nov 2 '18 at 19:27
• @whuber: I mean every observation has two attributes that are identical. Think copying a column in the data table. – tddevlin Nov 2 '18 at 20:46
• That's an example of collinearity. Ordinarily one would just drop one of the columns on the theory that they represent exactly the same property of the observations. (They might not, but as far as the data are concerned, they do.) – whuber Nov 3 '18 at 0:20