In section 4.6.2 (page 156) of An Introduction to Statistical Learning, the authors use logistic regression to predict daily stock price (up or down). First they use all 6 predictors and get a test accuracy of 0.52 (page 160); then they remove the predictors with high p-values, and fit a new model with the best 2 predictors and get a test accuracy of 0.58 (page 160).
They state:
Perhaps by removing the variables that appear not to be helpful in predicting Direction, we can obtain a more effective model. After all, using predictors that have no relationship with the response tends to cause a deterioration in the test error rate (since such predictors cause an increase in variance without a corresponding decrease in bias), and so removing such predictors may in turn yield an improvement.
But I thought logistic regression takes care of unhelpful predictors automatically through shrinkage via the lasso or ridge penalty? I.e. I thought one could have p >> n where a large number of p are just noise; and the model would shrink their coefficients to 0 (lasso) or near zero (ridge). So why does their test accuracy improve when they "manually" remove 4 predictors with high p-values?
