33
$\begingroup$

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a reasonable sense of the relationship between the variance in the model and large coefficients, I don't have as good a sense as to why they occur in overfit models. Is it incorrect to say that they are a symptom of overfitting and coefficient shrinkage is more a technique for reducing the variance in the model? Regularization via coefficient shrinkage seems to operate on the principle that large coefficients are the result of an overfitted model, but perhaps I'm misinterpreting the motivation behind the technique.

My intuition that large coefficients are generally a symptom of overfitting comes from the following example:

Let's say we wanted to fit $n$ points that all sit on the x-axis. We can easily construct a polynomial whose solutions are these points: $f(x) = (x-x_1)(x-x_2)....(x-x_{n-1})(x-x_n)$. Let's say our points are at $x=1,2,3,4$. This technique gives all coefficients >= 10 (except for one coefficient). As we add more points (and thereby increase the degree of the polynomial) the magnitude of these coefficients will increase quickly.

This example is how I'm currently connecting the size of the model coefficients with the "complexity" of the generated models, but I'm concerned that this case is to sterile to really be indicative of real-world behavior. I deliberately built an overfitted model (a 10th degree polynomial OLS fit on data generated from a quadratic sampling model) and was surprised to see mostly small coefficients in my model:

set.seed(123)
xv = seq(-5,15,length.out=1e4)
x=sample(xv,20)
gen=function(v){v^2 + 7*rnorm(length(v))}
y=gen(x)
df = data.frame(x,y)

model = lm(y~poly(x,10,raw=T), data=df)
summary(abs(model$coefficients))
#     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
# 0.000001 0.003666 0.172400 1.469000 1.776000 5.957000


data.frame(sort(abs(model$coefficients)))
#                                   model.coefficients
# poly(x, 10, raw = T)10                  7.118668e-07
# poly(x, 10, raw = T)9                   3.816941e-05
# poly(x, 10, raw = T)8                   7.675023e-04
# poly(x, 10, raw = T)7                   6.565424e-03
# poly(x, 10, raw = T)6                   1.070573e-02
# poly(x, 10, raw = T)5                   1.723969e-01
# poly(x, 10, raw = T)3                   6.341401e-01
# poly(x, 10, raw = T)4                   8.007111e-01
# poly(x, 10, raw = T)1                   2.751109e+00
# poly(x, 10, raw = T)2                   5.830923e+00
# (Intercept)                             5.956870e+00

Maybe the take-away from this example is that two thirds of the coefficients are less than 1, and relative to the other coefficients, there are three coefficients that are unusually large (and the variables associated with these coefficients also happen to be those most closely related to the true sampling model).

Is (L2) regularization just a mechanism to diminish the variance in a model and thereby "smooth" the curve to better fit future data, or is it taking advantage of a heuristic derived from the observation that overfiited models tend to exhibit large coefficients? Is it an accurate statement that overfitted models tend to exhibit large coefficients? If so, can anyone perhaps explain the mechanism behind the phenomenon a little and/or direct me to some literature?

$\endgroup$
  • 4
    $\begingroup$ What exactly do you mean by a "large" coefficient? After all, if we merely change the units in which we express the dependent variable (such as from parsecs to femtometers) we can make the coefficients arbitrarily large or small in value. $\endgroup$ – whuber Jul 13 '13 at 2:09
  • 1
    $\begingroup$ I don't have a good answer for that. My understanding was that attacking "large" coefficients was a motivating heuristic behind L2-regularization. But synthetically increasing the magnitude of the coefficients would require also changing the regularization constant to compensate for the different magnitude in the model now, wouldn't it? I don't think the notion of "large" here is as ambiguous as you're making it seem, even if I can't characterize it very well. $\endgroup$ – David Marx Jul 13 '13 at 2:16
  • $\begingroup$ @DavidMarx: I don't think that L2-regularization goes after "large" coefficients, does it? Rather it tends to push coefficients that probably weren't comparatively large to zero, in a sense forcing you to choose rather than find a compromise between the two. $\endgroup$ – Wayne Jul 13 '13 at 3:24
  • $\begingroup$ @wayne ah, I think I had it backwards. I had thought that it shrunk the larger coefficients or shrunk all coefficients proportionally. That would make more sense if L2-regularization pushed the variables with smaller coefficients out of the model. $\endgroup$ – David Marx Jul 13 '13 at 5:29
  • 1
    $\begingroup$ After 8 edits, I think I have my answer down. Sheesh. $\endgroup$ – Hong Ooi Jul 13 '13 at 19:08
15
$\begingroup$

In the regularisation context a "large" coefficient means that the estimate's magnitude is larger than it would have been, if a fixed model specification had been used. It's the impact of obtaining not just the estimates, but also the model specification, from the data.

Consider what a procedure like stepwise regression will do for a given variable. If the estimate of its coefficient is small relative to the standard error, it will get dropped from the model. This could be because the true value really is small, or simply because of random error (or a combination of the two). If it's dropped, then we no longer pay it any attention. On the other hand, if the estimate is large relative to its standard error, it will be retained. Notice the imbalance: our final model will reject a variable when the coefficient estimate is small, but we will keep it when the estimate is large. Thus we are likely to overestimate its value.

Put another way, what overfitting means is you're overstating the impact of a given set of predictors on the response. But the only way that you can overstate the impact is if the estimated coefficients are too big (and conversely, the estimates for your excluded predictors are too small).

What you should do is incorporate into your experiment a variable selection procedure, eg stepwise regression via step. Then repeat your experiment multiple times, on different random samples, and save the estimates. You should find that all the estimates of the coefficients $\beta_3$ to $\beta_{10}$ are systematically too large, when compared to not using variable selection. Regularisation procedures aim to fix or mitigate this problem.

Here's an example of what I'm talking about.

repeat.exp <- function(M)
{
    x <- seq(-2, 2, len=25)
    px <- poly(x, 10)
    colnames(px) <- paste0("x", 1:10)
    out <- setNames(rep(NA, 11), c("(Intercept)", colnames(px)))
    sapply(1:M, function(...) {
        y <- x^2 + rnorm(N, s=2)
        d <- data.frame(px, y)
        b <- coef(step(lm(y ~ x1, data=d), y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10, trace=0))
        out[names(b)] <- b
        out
    })
}

set.seed(53520)
z <- repeat.exp(M=1000)

# some time later...
rowMeans(abs(z), na.rm=TRUE)

(Intercept)          x1          x2          x3          x4          x5          x6          x7          x8          x9         x10 
   1.453553    3.162100    6.533642    3.108974    3.204341    3.131208    3.118276    3.217231    3.293691    3.149520    3.073062 

Contrast this to what happens when you don't use variable selection, and just fit everything blindly. While there is still some error in the estimates of $\beta_3$ to $\beta_{10}$, the average deviation is much smaller.

repeat.exp.base <- function(M)
{
    x <- seq(-2, 2, len=25)
    px <- poly(x, 10)
    colnames(px) <- paste0("x", 1:10)
    out <- setNames(rep(NA, 11), c("(Intercept)", colnames(px)))
    sapply(1:M, function(...) {
        y <- x^2 + rnorm(N, s=2)
        d <- data.frame(px, y)
        b <- coef(lm(y ~ ., data=d))
        out[names(b)] <- b
        out
    })
}

set.seed(53520)
z2 <- repeat.exp.base(M=1000)

rowMeans(abs(z2))
(Intercept)          x1          x2          x3          x4          x5          x6          x7          x8          x9         x10 
   1.453553    1.676066    6.400629    1.589061    1.648441    1.584861    1.611819    1.607720    1.656267    1.583362    1.556168 

Also, both L1 and L2 regularisation make the implicit assumption that all your variables, and hence coefficients, are in the same units of measurement, ie a unit change in $\beta_1$ is equivalent to a unit change in $\beta_2$. Hence the usual step of standardising your variables before applying either of these techniques.

$\endgroup$
  • $\begingroup$ I'm a little confused by your modified example. You say that "You should find that all the estimates of the coefficients β3 to β10 are systematically too large, when compared to not using variable selection," but it appears you got larger values in your first experiment (with step) than in your second experiment ("blindly" fitting values). Isn't this contrary to what you were suggesting should happen? $\endgroup$ – David Marx Jul 14 '13 at 15:43
  • $\begingroup$ Also, you and others here have suggested I should standardize the variables in my example. I see the reasoning, but I don't know of a good way to do that. Should I enhance my sample data to include columns for each power of X and standardize those values? Or is there a way I can standardize the variables directly in my model formula where I call poly (I'm guessing not)? $\endgroup$ – David Marx Jul 14 '13 at 15:45
  • $\begingroup$ ? You get larger absolute deviations when using stepwise, compared to not using stepwise. Not sure what you're asking. As for standardisation: it's unnecessary if you do it as I posted, ie making a comparison between the stepwise and non-stepwise approach. Each variable is being compared from one treatment to the other, rather than to the other variables. $\endgroup$ – Hong Ooi Jul 14 '13 at 15:57
6
$\begingroup$

One very simple answer without looking into your details: When you are overfitting, the parameter estimators tend to get large variances, and with large variances large values are just what you should expect!

$\endgroup$
  • $\begingroup$ If I understand you correctly, this would explain why the model would predict "large" values, not why the model would be comprised of "large" coefficients. $\endgroup$ – David Marx Jul 13 '13 at 18:26
  • $\begingroup$ No, that is wrong! At least some of the individual coefficient estimators will have large variances, so the estimated values of those coefficients will tend to be large. (of ourse, even when overfitting, some coefficients could be stable, but not all). Also, to keep the unbiasedness property of prediction, there will tend to be some large negative covariances between independent coefficient estimators. $\endgroup$ – kjetil b halvorsen Jul 13 '13 at 18:51
  • 1
    $\begingroup$ This doesn't fully answer the question, though. If it was just an issue of large variances, then you'd get small estimates just as often as large ones (informally speaking). The problem is when we then assume small estimates (relative to their std errors) are "unimportant" and drop those variables from the model. As a result, the only estimates remaining will be the large ones. $\endgroup$ – Hong Ooi Jul 14 '13 at 16:00
0
$\begingroup$

David. I think the problem with your example is you haven't normalised your data ( ie X^10>> X.

So david is right that it shrinks larger coefficients more ( so you can end up with lots of small coefficients, whilst L1 regularisation might give you one large and the rest zero)

so basically it is encapsulating that small changes should have small effects (and of course we are back to the issue of how small is small - normalising your data etc). But the key thing is in higher dimensions, where correlation comes into play: imagine you have two variables x,y that are highly correlated (both normalised to variance 1) then their difference will be small ="noise" - penalising large weights will therefore prevent you fitting to this noise ( and getting very large almost cancelling coefficients for y and x).

The example still holds for any linear relation (y=mx)

look up ridge regression

$\endgroup$
  • 1
    $\begingroup$ david why don't you redo the example normalising all the variables x,x^2, ...,x^n to zero mean and standard deviation 1, and then see the results you get... its not very surprising that your coefficients are small when your input variables are large $\endgroup$ – seanv507 Jul 13 '13 at 20:56
0
$\begingroup$

overfitting demo

This image is from my note of Andrew Ng's DL course, pls let me know if you have question

$\endgroup$
  • 1
    $\begingroup$ Could you explain why you think this note on neural network regularization answers the question about coefficient sizes and overfitting? $\endgroup$ – whuber Apr 23 at 5:02
  • $\begingroup$ all 3 are linked overfitting is just what regularization tries to prevent and it tries by penalizing high coefficient due to the reason above $\endgroup$ – Pradi KL Apr 24 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.