There is a nice answer, however it goes from another way around: the model gets more bias if we drop some features by setting the coefficients to zero. Thus, overfitting is not happening.

I am interested more in my large coefficients indicate the overfitting. Lets say all our coefficients are large. My intuition is that the larger coefficients get the less important are the features. However, if this is the case how are we able fit too much to the data if features are less important?

  • 1
    $\begingroup$ Could you explain that intuition? It would seem to many people, on the contrary, that large coefficients correspond to features that have strong relationships with the response. Please note that we must understand the sense of "large" by assuming all variables have been standardized; otherwise, the size of a coefficient has no inherent meaning. $\endgroup$ – whuber May 18 '17 at 22:09
  • 1
    $\begingroup$ I was thinking if we change the units of out features, the coefficients might get larger (or smaller) but the predictions will be unaffected. So, large coefficients does not mean that features are important. However, if we standardize the features a large coefficient can mean that the feature is important, can't it? $\endgroup$ – Alina May 18 '17 at 22:22
  • 4
    $\begingroup$ Changing the units of the features does not change the standardized features at all, and therefore is irrelevant. With standardized features, a larger coefficient means a larger change in the response is associated with a unit change in the feature on the scale used to standardize the feature (usually a standard deviation scale). $\endgroup$ – whuber May 18 '17 at 22:29
  • $\begingroup$ Are you thinking of situations where collinearity causes two coefficients to get very large but of opposite signs? $\endgroup$ – Wayne May 18 '17 at 23:40
  • $\begingroup$ @whuber lets say we standardized our features and got large coefficients. why is it a sign of overfitting? Is it because by changing a feature by one unit there is a large change in the response, meaning with this coefficient we artificially induce the exact fir to each point? $\endgroup$ – Alina May 19 '17 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.