Questions tagged [lasso]

A regularization method for regression models that shrinks coefficients towards zero, making some of them equal to zero. Thus lasso performs feature selection.

470 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
14
votes
1answer
20k views

Standardization vs. Normalization for Lasso/Ridge Regression

I am aware it is common practice to standardize the features for ridge and lasso regression, however, would it ever be more practical to normalize the features on a (0,1) scale as an alternative to z-...
14
votes
0answers
468 views

Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
9
votes
0answers
491 views

Can Frank Harrell's method be used to obtain optimism-corrected regression coefficients?

I used a regularized (LASSO) Cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. ...
7
votes
0answers
1k views

Selecting regularization penalty: cross validation or information criteria?

I will use an elastic net to estimate a regression model which will later be used for forecasting. I have a grid of $\alpha$ values within [0,1] representing the proportion of $L_1$ versus $L_2$ ...
7
votes
0answers
722 views

Lasso for GEE model

Can a LASSO be applied for predictor selection in a logistic GEE (generalized estimating equations) model for longitudinal data? Is there an implementation of LASSO for a logistic GEE model for ...
7
votes
0answers
900 views

Dantzig Selector, LASSO, LAD LASSO

I am wondering about this. When is it best to use Dantzig Selector (the infinity normed error measure plus the L1 regularizer) , the LASSO (the mean square error measure plus the L1 regularizer), and ...
6
votes
1answer
165 views

Asymptotic bias of LASSO vs. none of SCAD

I am reading a paper which says that LASSO is asymptotically biased while SCAD is not. I take asymptotic (un)biasedness to concern the slope estimators from LASSO and SCAD as the sample size goes to ...
6
votes
0answers
833 views

Geometrical interpretation of L1 regression

I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
6
votes
0answers
654 views

Is lasso always outperformed by adaptive lasso?

I have been reading some papers and I understood that adaptive lasso has the Oracle properties which lasso lacks. Does that mean adaptive lasso always better than lasso (let's focus on the simple ...
6
votes
0answers
1k views

When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
5
votes
1answer
101 views

Once you have used LASSO to generate regression coefficients, is there another step that gives you information about the model?

I've run a LASSO to build a model out of ~60 potential predictors. I'm wondering what the next step is? If there were OLS regression I would find model fit statistics like R2 or AIC. I would also find ...
5
votes
1answer
944 views

L1 and L2 regularization showing increased MSE with added vars (that eventually decreases)

I am attempting to run Ridge, LASSO, and Elastic Net regression as the regularization approaches are commonly used in the problem I'm working to solve. I have successfully run both glmnet() and cv....
5
votes
0answers
50 views

Before using CV-selected Regression model for Inference, shouldn't model performance be evaluated on unused test set?

I just came across a biokinesiology paper that used some Machine Learning methods, but I think there is a flaw in their methodology. The authors had data on stroke patients and used Lasso regression ...
5
votes
0answers
718 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
5
votes
0answers
965 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
5
votes
0answers
2k views

What is the difference between Lasso regression in glmnet (in R) and Sklearn lasso (in Python)?

A similar post was discussed here regarding Ridge Regression: What are the differences between Ridge regression using R's glmnet and Python's scikit-learn? My question is what is this ...
5
votes
0answers
880 views

How to group continuous variables in LASSO for a multinomial regression model?

With the goal of selecting predictors for a 4 level outcome variable I want to apply LASSO for predictor selection. Some continuous variables are related to each-other and should all be in the final ...
5
votes
0answers
405 views

Prior for $\lambda$ is LASSO prior?

I have a regression model with regression coefficients $\beta_j$, $j=1,...,n$, and I would like to use a LASSO prior for $\beta_j$, this is: $$\beta_j \sim Laplace(0,1/\lambda),$$ where the Laplace ...
5
votes
1answer
2k views

T-test for regression coefficients obtained from Ridge, LASSO etc

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by: $ \hat B = {({X^t}{X})}^{-1}X^ty$ Then, my null hypothesis for each coefficent is: $...
5
votes
0answers
301 views

Consistency of variable selection based on Lasso estimator for high-dimensional data

According to this paper by Meinshausen and Bühlmann (2006) the variable subset selection coming out of a Lasso is not always consistent in high-dimensional cases. It is bounded by the neighbourhood ...
5
votes
0answers
367 views

What is least angle regression?

Conceptually, I don't understand what least angle regression Least Angle Regression (LARS) is and why it solves LASSO (pdf). We know that LASSO is: $$\arg \min_x {\left\| A x - y \right\|}_{2}^{2} +...
5
votes
0answers
228 views

Running regularized logistic regressions on very large datasets

I want to run a regularized logistic regression on a dataset with 25 million observations and about a 1000 mostly non-sparse columns with non-ignorable weights. My first choice would be BayesGLM, ...
4
votes
0answers
48 views

Asymptotic recovery of sparse coefficient vector with lasso

Given $\beta^* \in \mathbb{R}^d$, suppose $X_n \sim N(\mathbf{0}, I_d)$ are iid and $Y_n = X_n\beta^* + \epsilon_n$, where $\epsilon_n \sim N(0, 1)$ and are iid. Let $\hat{\beta}_n$ be the solution to ...
4
votes
0answers
83 views

Do variable-selection methods (e.g. Elastic Net; Lasso) invalidate theory-based models in fields where little is known?

I'm caught in a bind about the relationship between theoretical models about how the world works and statistical methods for accurately predicting an outcome in fields where little is known. I ...
4
votes
0answers
155 views

How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression?

How to prove the equivalence between constrained form and Lagrange form for lasso and ridge regression? Given lasso (constrained form): $$\underset{\beta}{\min}{(\frac{1}{2N}||y-x\beta||_2^2)} \...
4
votes
0answers
423 views

How to consider interaction terms in the ridge / lasso / elastic net?

I would like to ask a question about how to consider interaction terms in my penalized regression? My primary goal is to build the model to predict. I think in the conventional GLM, we run the model ...
4
votes
0answers
737 views

Is it correct to use the model provided by LASSO to predict an outcome?

I have 214 covariates and a binary outcome. The total number of positive and negative outcomes is 27 and 33, respectively. I already modelled it using a 1v1 univariable logistic regression and I ...
4
votes
0answers
128 views

Hyperprior for Bayesian LASSO & Horseshoe

I currently sample LASSO and Horseshoe regression in STAN. Hence I was wondering how to properly define the hyperpriors in the bayesian regression models. I.e. Park and Casella use a gamma prior with ...
4
votes
0answers
108 views

Incorrect computation in Knight and Fu (2000)?

I'm currently reading Knight and Fu's 2000 paper on the asymptotics of "Bridge" estimators with a particular focus on LASSO as a special case. In the proof of theorem 2, they make the claim that under ...
4
votes
0answers
93 views

lambda in lasso regression

How can I show that the smallest value of λ such that the regression coefficients estimated by the lasso are all equal to zero is given by $$ λ = max_j \left| \left< x_j, y \right> \right| $$
4
votes
0answers
103 views

lasso fails with few large effects and many small effects

I have an application that is predicting height based on sex and DNA mutations. As height is very different depending on sex, sex is a variable with a strong effect in my prediction (~13). On the ...
4
votes
0answers
114 views

Dealing with many NA's in very large datasets for Lasso

I have a few very large and quite "dirty" (survey) datasets. Primarily, there are lots of NA's. These NA's are mostly the result of different questions being asked in different waves. It is ...
4
votes
0answers
171 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
4
votes
0answers
80 views

What is the differences between LASSO and SALSA?

When I see their formulation, it is the same. In SALSA, the formulation is: \begin{equation*} \min_{x} \phi(x) \text{ subject to } \frac{1}{2} \|Ax-y\|_F \leq \epsilon \end{equation*} This ...
4
votes
1answer
102 views

Lasso on squared parameter

Assume a linear regression problem where I want to force sparsity of some parameters. However, due to some physics, I know that one of my parameters is always positive. For instance, I have that $$ ...
4
votes
0answers
350 views

Proximal Gradient Descent and Proximal Coordinate descent for Lasso Problem

Why is proximal coordinate descent much less affected by bad conditioning than proximal gradient descent? For example, we can consider this problem : $\min_x \frac{1}{2}\|Ax-b\|^2_2 + \lambda\|x\|_1$ ...
4
votes
0answers
354 views

The method of knock-offs by Barber & Candes for variable selection and FDR control

The knock-off method is a recent approach to variable selection and FDR control presented in two papers to be found here https://statweb.stanford.edu/~candes/papers/FDR_regression.pdf and here https://...
4
votes
0answers
193 views

Using Bayesian Lasso with an informed prior

I'm looking for advice on how best to go about setting an informative prior for the Bayesian Lasso and BART (I'm applying these in R using the rjags and bartMachine packages) I have 3 proteomics ...
4
votes
0answers
117 views

Multivariate Regression equivalent to Multinomial Regression (aggregated variables)?

Context (optional) In genetics one often uses data coming from SNP (Single Nucleotide Polymorphismes) which are genetic markers of which several (usually 2) versions (a.ka. alleles) exists in a given ...
4
votes
0answers
784 views

MSE of an individual coefficient from ridge or lasso vs. OLS

Consider a multiple regression model $$ y = X\beta + \varepsilon $$ with $K$ regressors in $X$. If the model is correctly specified, the OLS estimator $\hat\beta_{OLS}=(X'X)^{-1}X'y$ will be the ...
4
votes
0answers
175 views

Variability in LASSO models for predicting rare events

I want to build a model for predicting a rare (ca 10%) event in my dataset of around 300 samples and 15 candidate predictors (of these, I know that five, when looked at individually in the whole ...
4
votes
0answers
1k views

R - quadprog package for constrained Lasso (penalized) linear regression

What I am doing so far: I am doing a constraint linear regression with R's quadprog package, function solve.QP(). The ...
4
votes
0answers
768 views

Why does this multi-response Guassian LASSO not give a sparse solution?

I tried the glmnet package to learn multi-response Gaussian family. I have looked at the coefficients of the final model. The result is odd. All the features have ...
4
votes
0answers
1k views

Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
4
votes
0answers
1k views

GLS, heteroskedasticity and Ridge Regression/Lasso

I am hoping to use a regularised regression technique, using cross validation, to fit a linear model to a set of predictors which have some highly correlated variables. However, I also know (highly ...
4
votes
0answers
521 views

Standardize continuous predictor variables on [0, 1] scale?

I'm working on a health care regression model predicting # of inpatient visits. My analysis dataset includes a number of hybrid continuous/categorical predictor variables which can hold values on a 0 ...
4
votes
0answers
279 views

Logistic regression model for analysis of many IVs with a relatively small sample size

I'm trying to determine the influence (direction and relative strength) of certain attributes of incoming students to an academic program on their successful completion of the program. My sample size ...
4
votes
0answers
673 views

Kernel in PenalizedSVM R package

There is not option to select kernel in penalizedSVM R package. What kernel do they use? Is there some other R package with penalized SVM methods where I can choose various kernels?
3
votes
0answers
57 views

Does it make sense to deal with multicollinearity prior to LASSO regression?

Does it ever make sense to check for multicollinearity and perhaps remove highly correlated variables from your dataset prior to running LASSO regression to perform feature selection? One of the ...
3
votes
0answers
126 views

How to generate binary outcome data where ridge or lasso outperforms simple logistic regression?

The answer is very easy to the question in the title, we just need a data set where $n<p$; thus, logistic regression MLE does not exist. Also if perfect separation occurs, ridge or lasso will be ...

1
2 3 4 5
10