what is the idea behind SHRINKAGE (regularization) METHOD (e.g LASSO), and how to interpret it?

Question

I've read ALOT of papers and questions here as well, on other websites, but I didn't get the point behind the LASSO. my background is in economics (not an expert in math).

sorry for my bad english...

so, broadly speaking our desired total cost function by which we want to balance between

1. how well function fits data. (the RSS)
2. Magnitude of coefficients. ( by forcing some restriction either l2 or l1( here comes the main problem)

my first question is why would we want to reduce the magnitude of coefficients? what is the idea behind it? for example (arbitrary one), if I want to explain (or whatsoever) the DV through some explanatory variables why would I prefer the estimated coefficients to be less in terms of their magnitude ( I know they should be standardized).

my second question is, what is the criteria behind the lasso optimization? in other words when the lasso omits some variables from the model how did he ( lasso) remove them? (I know it's due to the diamond shape of the constraints).

I understand that the lasso constraint has corners and so the ellipse will often intersect the constraint region and when this occurs, one of the coefficients will equal zero.

for example, the criteria behind stepwise or best subset are well understood ( e.g some factors are less correlated than others with the DV, so they wipe off from the model. (I just explain the very big idea so you can understand me well). another example, when we use PCA the general procedure is easy to understand, we project each data point onto only the first few principal components to obtain lower-dimensional data and so on.. so again the logic behind it the easy to understand.

in a few words, I understand how stepwise omits some variables based on some criteria. in contrast, I can't understand why some variables will be omitted from the model (they touch that diamond but why would they in the first place touch it? based on what? so, for example, if I have 10 independent variables, and the lasso omits 4 of them, are they insignificant in terms of p-values? are they making MSE larger? what is the basic explanation if I want to interpret why they dropped off from the model?

Answer 1

Why omit some variables is an excellent question. Many analysts do not realize that the data seldom contain sufficient information to correctly decide on which predictors to omit, thus variable selection has too much randomness and the probability of selecting the right variables, whether using lasso, elastic net, or other methods, may be close to zero.

To the question of why shrink coefficients, as explained here if you select a predictor at random its coefficient estimate will be unbiased, but we don't select predictors for examination at random; we look for "winners", and this creates a selection process that exaggerates coefficients. And if you sorted predicted values in ascending order you'd find that the lowest predictions are too low and the highest are too high. Shinkage will lessen that effect.

lasso removes predictors by using cross-validation to pretend that we can choose a penalty parameter optimizing some criterion, and large penalties remove more predictors.