# Tag Info

4

I realize that when I use LASSO to select a model, I can't infer from it because it penalizes the coefficients to best fit the data. I'm just a little confused as to what exactly this means. Isn't this also what a regression does? It tries to fit the data best possible? If I can't infer from a LASSO model, what do I use it for? LASSO is like OLS plus ...

3

A function is convex if it obeys the following inequality, for $\{0<a<1, a+b=1\}$: $$f(ax+by)\leq af(x)+bf(y)$$ Restricting the inequality we can define strict convex functions as the ones that obey $$f(ax+by)\lt af(x)+bf(y)$$ If $f(x) = \sqrt{x^2}=|x|$, then by the triangle inequality: $$f(ax+by)=|ax+by|\leq a|x|+b|y|$$ Thus the absolute value is a ...

2

You should use group lasso. When you have a categorical variable, with say, $k$ levels, usually represented in a regression model with $k-1$ dummys, that group of $k-1$ dummys together is the one variable, and should be included or not as a unit. Group lasso does that. This have been explained many times on this site, for example Can I ignore coefficients ...

2

We don't generally consider $\lambda$ as a parameter in the model you want to estimate. It doesn't have an interpretation outside of the model, in terms of your actual data. Instead, we consider $\lambda$ as a tuning parameter or a hyperparameter. This terminology means that $\lambda$ affects how you estimate $\beta$, but you aren't interested in $\lambda$ ...

1

One big problem here is the combination of LASSO with a small number of cases. LASSO at its optimum penalty factor will typically retain a number of predictors close to the number that can be comfortably be assessed without overfitting. In a logistic regression, that's about 1 predictor for every 15 or so members of the minority class. With only 51 cases ...

1

Is my strategy of forward selecting the features according to the mean CV AUC OK here? Since you are using Python you should be aware of Recursive Feature Selection CV which in essence is doing what you are, but this gives you the advantage of working with pipelines Is it OK to keep the LASSO regularization (which also select feature), in the loop? This ...

1

The usual practice in penalized methods like LASSO, ridge or principal-component regression is to start by standardizing all predictors to zero mean and unit standard deviation. Otherwise the issue that you raise would lead to a distance measured in miles being handled differently than the same distance measured in millimeters. That's the default, for ...

1

Are some features highly correlated ? According to this paper about elastic net: Lasso variable selection is poor when the number of feature is higher than the number of observations. Lasso variable selection is unstable when you have groups of features that are highly correlated.

Only top voted, non community-wiki answers of a minimum length are eligible