# Tag Info

-1

LASSO is a procedure that pushes some of the parameters which have small effect to be zero. It can thus be used to select the best subset of variables to predict your response. You can eg use these variables in a least square fit afterwards, insteaf of the full set of parameters

1

Ridge was originally designed for correlated variables, and that's where it's best. Consider examinations to determine a degree. ( Which supposedly is measuring ability) Which do you think is more reliable: taking the average of all the exams or picking a single exam most correlated with ability (if there is one)? Averaging over the different exams removes ...

6

Both models penalise the inclusion of a non-zero coefficient, using a penalty function. LASSO regression penalises in a way that is proportionate to the absolute magnitude of the coefficient, and ridge regression penalises in a way that is proportionate to the square of the coefficient. Neither model penalises inputs in the likelihood function where the ...

1

For prediction, we are interested in the best outcome and want to include as much information as possible in the model to explain the response, but still without over fitting (don't capture the noise) as we want our model to generalize well to new data. Generally, lower values of the LASSO tuning parameter are needed for prediction. When a group of ...

0

another issue is that regression is often used to control for effects of other variables. Lets say I want to know if A is related to Y controlling for B, A and B are strongly correlated and my answer is no, however if I regularize A and B coefficients then my answer will be yes, which is wrong.

1

The y-intercept will be the last coefficient in your weight vector, in the case where fit_intercept=True. Not sure about the .add_constant(x) function but if it concatenates a list of 1s to your dataframe, then yes, it is the same as the fit_intercept parameter. In this case, you will want to check if it's doing pd.concat([data,ones],'1') or pd.concat([ones,...

-1

It may be an excellent idea to run an OLS regression after LASSO. This is simply to double check that your LASSO variable selection made sense. Very often when you rerun the model using OLS regression you uncover that many of the variables selected by LASSO are nowhere near being statistically significant and/or have the wrong sign. And, that may invite ...

0

First, I would not double up on variable selection methods. This does not make much sense. I would instead use the best method given your data and model framework, and go with that. Second, there are numerous reasons to be very cautious regarding using LASSO as your preferred variable selection method. There is a simple reason why not using LASSO for ...

-1

There is a simple reason why not using LASSO for variable selection. It just does not work as well as advertised. This is due to its fitting algorithm that includes a penalty factor that penalizes the model against higher regression coefficients. It seems like a good idea, as people think it always reduces model overfitting, and improves predictions (on ...

0

The straightforward answer is that LASSO does not always work as intended. Given its penalty factor LASSO has no way of differentiating between a true causal variable that has a high coefficient and should be selected in your model vs. another variable that has little relationship with Y and has a low coefficient. The LASSO algorithm may randomly and often ...

4

Maximum Likelihood just like minimizing the square of the sum of residuals has no penalty factor to reduce coefficients to zero as LASSO does. So by definition Maximum Likelihood does not shrink coefficients to zero and LASSO does. However, you can't automatically deduce that the LASSO model will generate a better fit and prediction of what you are ...

4

I'll add this: That the LASSO (and associated with it, ridge regression) have Bayesian interpretations is a glimmer of light in the problem of penalization and inference. However, it isn't as easy as "pick priors so that some covariates are shrunk to zero". In order for valid inferences to be made from a bayesian linear model, your priors have to actually ...

3

First of all, the LRT is not really a good measure for variable selection. It is intended to compare nested models for evaluation of a hypothesis. If you use the LRT to do variable selection for 10 variables, then you are almost certainly going to make a false positive. If you want to do a form of variable selection, LASSO seems to be the way to go and ...

4

A danger in having powerful tools available in standard software programs is that users don't always understand the underlying hidden assumptions. Even if you will be using Stata for routine work, I recommend getting a copy of An Introduction to Statistical Learning and working through the examples in Chapter 6 of LASSO and ridge regression, with the code ...

2

I'm a little confused about what you mean. Lasso is a specific type of regression algorithm that given some input, produces output predictions. It has nothing to do with splitting data — you have to do that manually.

3

This distinction isn't always pre-specified in software, but it is often available (although this distinction might not be the default so that you have to read the manual to figure out how to do it). It's a way to limit regularization to predictors that control for additional influences on outcome while not penalizing coefficients of the predictors of ...

5

You can do it no matter where the parameter is. For instance, this is done in neural networks, where the layer weights are highly nonlinearly related to the target value. In general, when the parameters are linearly related to the target, you just have an advantage on parameter scaling so that regularization objective treats each one equally, when feature ...

2

The importance measures here are unlikely to be adapted into a scale that can be compared. Rather, the best way to compare these models is to compare the rank order of the input/predicting variables. It is also worth noting that the linear/lasso/ridge models are fairly different in what they represent compared to boosted tree ensemble. In particular, the ...

0

The definition I know, is close to what is described in this preprint (Schad et al, arxiv:1904.12765): posterior contraction estimates how much prior uncertainty is reduced in the posterior estimation: $$s = 1 - \frac{\sigma^2_{post}}{\sigma^2_{prior}}$$ Here, the variance of the posterior distribution, $\sigma^2_{post}$, is divided by the ...

1

Posterior contraction is about the contraction of the posterior distribution (of some random variable of interest), not the posterior mode. Under Bayesian models that obey standard regularity conditions, as data are observed, the posterior distribution of a random variable morphs from the prior distribution to a point mass (Dirac measure) centered at the ...

1

Judging by the data that you present, it does not look like an inputation problem, but rather a prediction one. If I am not mistaken, you have a first representative dataset containing your target variable, and a second dataset where you want to insert it. This can be seen as a supervised learning problem (and indeed, LASSO is a supervised learning algorithm)...

3

People often assume that regularization is superior to un-regularized models because they reduce multicollinearity, reduce model overfitting, and improve forecasting. They also like regularization because it explicitly avoid the entire body of model testing associated with the Gauss-Markov Theorem and other related underlying assumptions of regular ...

0

My comment would be that all boils down to assumptions. While we would like a hard and fast rule to everything, the world is at the very least a little more complicated than this. Blindly applying either is bound to mislead our interpretation. While we cannot test if the data fits every possible model or assumption, nor should we, we can get into ...

0

Lasso does neither of those things. It forces the coefficients of the least important predictors to be zero, but not necessarily the intercept (this would be problem dependent). At the same time, since the loss function contains the absolute value of the coefficients, their sign is irrelevant and only their magnitude has an effect on the result.

27

In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot. In instances where we want to use regression to make inferences about interventions, we want unbiasedness. Not everything to do with data is a prediction problem.

Top 50 recent answers are included