I've read ALOT of papers and questions here as well, on other websites, but I didn't get the point behind the LASSO. my background is in economics (not an expert in math).
sorry for my bad english...
so, broadly speaking our desired total cost function by which we want to balance between
- how well function fits data. (the RSS)
- Magnitude of coefficients. ( by forcing some restriction either l2 or l1( here comes the main problem)
my first question is why would we want to reduce the magnitude of coefficients? what is the idea behind it? for example (arbitrary one), if I want to explain (or whatsoever) the DV through some explanatory variables why would I prefer the estimated coefficients to be less in terms of their magnitude ( I know they should be standardized).
my second question is, what is the criteria behind the lasso optimization? in other words when the lasso omits some variables from the model how did he ( lasso) remove them? (I know it's due to the diamond shape of the constraints).
I understand that the lasso constraint has corners and so the ellipse will often intersect the constraint region and when this occurs, one of the coefficients will equal zero.
for example, the criteria behind stepwise or best subset are well understood ( e.g some factors are less correlated than others with the DV, so they wipe off from the model. (I just explain the very big idea so you can understand me well). another example, when we use PCA the general procedure is easy to understand, we project each data point onto only the first few principal components to obtain lower-dimensional data and so on.. so again the logic behind it the easy to understand.
in a few words, I understand how stepwise omits some variables based on some criteria. in contrast, I can't understand why some variables will be omitted from the model (they touch that diamond but why would they in the first place touch it? based on what? so, for example, if I have 10 independent variables, and the lasso omits 4 of them, are they insignificant in terms of p-values? are they making MSE larger? what is the basic explanation if I want to interpret why they dropped off from the model?