Timeline for Why do we need regularization for linear least squares given that a line is the simplest model possible?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 2, 2022 at 8:24 | comment | added | PascalIv | If you have a small w1, replacing the model with predicting the training mean does not introduce a large error. It is therefore closer to the simpler model of just using the mean. I think this all becomes more clear when looking at models with more than one parameter. If you regularize a linear model with many parameters using L2 regularization, it is less able to overfit complex noise patterns. The model is simpler, but parameters usually don't shrink to zero, so the number of parameters stays the same. | |
| Nov 1, 2022 at 16:57 | comment | added | Antonios Sarikas |
I can't get why large weights correspond to complex models. There is no reason that because w1 >> w2, y=w1*x is more complex than y=w2*x. The only way I can understand regularization is that large values of it will lead the model to predict the mean of the training data.
|
|
| Nov 1, 2022 at 9:57 | comment | added | PascalIv | @adosar You're absolutely correct. But extreme values like w=10000 just happen to be rare in systems we study. See it in a Bayesian way: The regularization term represents the prior knowledge, that the true model is simple (w is small). Given enough evidence/data, however, your model will converge to the true model. But extraordinary claims (w=extremly large) demand extraordinary evidence :) | |
| Oct 28, 2022 at 19:03 | comment | added | Antonios Sarikas |
What if your three data points where exactly on the red line? Why regularization would penalize those weights (for simple regression this means that maybe w=10000) more than if they were on blue or green line? Regularization will shrink w towards 0 but it seems that it does it only when w is too high. I mean if the true model is y=1x and we try to approximate it then we don't get then same penalty when the true model is y=10000000x and we try to approximate it (given the same value of regularization).
|
|
| Aug 25, 2022 at 9:43 | history | edited | PascalIv | CC BY-SA 4.0 |
added detail
|
| Aug 24, 2022 at 7:20 | history | answered | PascalIv | CC BY-SA 4.0 |