Case where Logistic Regression performs better with fewer predictors

In section 4.6.2 (page 156) of An Introduction to Statistical Learning, the authors use logistic regression to predict daily stock price (up or down). First they use all 6 predictors and get a test accuracy of 0.52 (page 160); then they remove the predictors with high p-values, and fit a new model with the best 2 predictors and get a test accuracy of 0.58 (page 160).

They state:

Perhaps by removing the variables that appear not to be helpful in predicting Direction, we can obtain a more effective model. After all, using predictors that have no relationship with the response tends to cause a deterioration in the test error rate (since such predictors cause an increase in variance without a corresponding decrease in bias), and so removing such predictors may in turn yield an improvement.

But I thought logistic regression takes care of unhelpful predictors automatically through shrinkage via the lasso or ridge penalty? I.e. I thought one could have p >> n where a large number of p are just noise; and the model would shrink their coefficients to 0 (lasso) or near zero (ridge). So why does their test accuracy improve when they "manually" remove 4 predictors with high p-values?

• What makes you think there is any shrinkage going on in the GLM fitted on page 156? "Generalized Linear Model" does not imply regularization. Commented Apr 22, 2018 at 1:08
• Thanks. Is there any reason to not use regularization if only prediction accuracy is of interest? Commented Apr 22, 2018 at 1:15
• @jbowman. You are correct. They are not using regularization. When I use shrinkage and select the tuning parameter alpha via cross validation, I get 0.56 accuracy when using all six predictors. Commented Apr 22, 2018 at 2:33
• The answer to this post stats.stackexchange.com/questions/281601/… might help with your question about not using regularization. Commented Apr 22, 2018 at 4:06

At least two points:

• A logistic regression (or more generally a generalized linear model) do not imply use of regularization, it is often fitted with mle (maximum likelihood estimation) without any regularization. And, regularization do not necessarily improve the fit.

• You try to evaluate the fitted model by accuracy, which is not a proper scoring rule. But it is fitted by optimizing another criterion (likelihood.) By including more variables, the criterion it is fitted on will improve (on the training set, that is), but on some other criterion, like accuracy, it does not need to improve. So there is no contradiction!

Also see the good comments on the question.