1
$\begingroup$

Given a vector of training data y and a corresponding matrix of features X (the ith row of X containing the feature vector associated with observation yi), why the training error (ie the mean squared error on the training set) of the least squares prediction will never increase as more features are added, according to linear algebra. Will the test error (ie the mean squared error on an independent test set) also never increase as more features are added ?

$\endgroup$

2 Answers 2

1
$\begingroup$

No; reducing a model's training error too aggressively can lead to test error increasing rather than decreasing. This phenomenon is called overfitting and is one of the primary obstacles to selecting good predictive models. Here's an example in R:

 set.seed(5)
 N = 1000
 d = data.frame(
     x1 = rnorm(N),
     x2 = rnorm(N))
 d = transform(d, y = x1 + rnorm(N))
 d.sample = d[1:5,]
 d = d[6 : N,]

 rmse = function(observed, predicted)
    sqrt(mean((observed - predicted)^2))

 m1 = lm(y ~ x1, data = d.sample)
 message("Training error: ", rmse(d.sample$y, predict(m1)))
 message("Test error: ", rmse(d$y, predict(m1, newdata = d)))

 m2 = lm(y ~ x1 + x2, data = d.sample)
 message("Training error: ", rmse(d.sample$y, predict(m2)))
 message("Test error: ", rmse(d$y, predict(m2, newdata = d)))

Adding the uninformative feature x2 decreases training error but increases test error.

$\endgroup$
0
$\begingroup$

Statistically,we can say that the added features are useless.But we have to verify with the domain experts whether should we consider those features,as it may give some useful trend.

Hope this helps!!!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.