Questions tagged [lasso]
A regularization method for regression models that shrinks coefficients towards zero, making some of them equal to zero. Thus lasso performs feature selection.
1,433
questions
0
votes
0
answers
9
views
Betas from precision matrix?
This is really 2 questions.
I would like to do a linear multi-regression on a large quantity of data, so large that I cannot really store it (it’s about 1e10 observations across 2500 features). Hence ...
- 1
0
votes
0
answers
16
views
How to model suspected piecewise-linear data with a lasso GLM
My data consist ~130 observations. Each observation has several thousand features (including many collinear or otherwise useless features) and a position along a single spatial dimension. Some sets of ...
- 63
0
votes
0
answers
11
views
Which standard error of out-of-sample prediction errors is used to select the model one standard error from the minimum?
In this post I established that the standard error of cross-validation prediction error is the standard deviation of prediction error across folds divided by the square root of the number of folds. ...
- 1,935
0
votes
0
answers
28
views
What is the standard error of average cross validation error in elastic net or lasso?
I don't have any colleagues who I can ask about this so I must turn to my colleagues on Cross Validated.
I am fitting a stacked adaptive elastic net regression and am having some trouble understanding ...
- 1,935
2
votes
0
answers
38
views
Distribution of Penalized Regression Coefficients
For both linear and logistic regression we know that the coefficient vector $\hat\beta$ holds an asymptotic normal distribution, therefore the the distribution of the linear predictor $\hat\theta_i=x^...
- 3,028
1
vote
0
answers
24
views
Proof of Variable Selection Consistency for LASSO in Zhao & Yu, 2006
I'm going through the proof of Proposition 1 in Zhao & Yu, 2006 (https://www.jmlr.org/papers/volume7/zhao06a/zhao06a.pdf), titled On Model Selection Consistency of LASSO. The proof is in Appendix ...
- 21
0
votes
0
answers
8
views
Lasso Regression - Extracting the penalty term
If I run a lasso regression model, and get a series of coefficients, say :
$\widehat{\beta _{1}},\widehat{\beta _{2}},...,\widehat{\beta _{p}}$
Is it legitimate to say that the value of the penalty ...
- 456
0
votes
1
answer
24
views
Which standardization for lasso and dependent variable
What kind of standardization of the predictor matrix is needed for lasso?
let y be the response variable
is it (x-mean(x)/sd(x)) OR (x-mean(x)/sd(y))?
Also does lasso require you to standardize the ...
- 1
0
votes
0
answers
32
views
Lasso coefficients and penalization impact
What is the correct interpretation of the coefficients from a lasso regression if you standardize ((x-mean(x))/sd(x) the predictor variables?
If you observer 4.2 for variable A and -1.2 for variable B ...
- 1
0
votes
0
answers
12
views
Optimal method for predicting outcome from additive, correlated, and sparse features?
Suppose I have many vectors which can take on any of three values, 0, 1, 2. These vectors affect an outcome being predicted, Y. Vectors add together: a vector "A" of the value 2 has twice ...
- 151
2
votes
0
answers
23
views
Coordinate Descent Alternating between LASSO and Ridge
Is there a way to do Coordinate descent but depending on the variable change the method applied to find the coefficient?
For example, apply a LASSO constraint to a predefined 3 variables and Ridge to ...
- 1,502
0
votes
0
answers
7
views
Compare LASSO regressions as missing data increases
I'm comparing three LASSO-regression models for classifying two patient types. Each model has increasingly complex variables, which are less likely to be available. Consequently, the last model has ...
- 39
0
votes
0
answers
26
views
Lasso for non-stationary time series
Does it make sense to use Lasso to find an explanatory variable x to predict my variable y, assuming both y and x are non-stationary? (I'm using both variables as levels, not differences). If I find a ...
- 1
0
votes
0
answers
16
views
LASSO shrinking variables RLM finds important
Comparing results from LASSO and robust regression I find that in some cases variables with high magnitude coefficients in robust regresison are shrunk to 0 in LASSO.
Any intuition here? Coefficients ...
- 171
1
vote
0
answers
74
views
Treatment of blocking variables in LASSO regression
By reviewing the existing relevant questions I could not find the answer to this specific question.
I have created blocking variables with the one-hot method (n - 1 binary variables for n categorical ...
- 11
0
votes
0
answers
37
views
What's the effect of doubling the data and copying the data for Lasso Regression and Ridge Regression on standard error?
Suppose we have a dataset $X$, where each piece data of $X$ is a row vector, and the data generation process satisfies Gaussian-Markov assumption. If we do ridge regression on $Y\sim 2X$, how does the ...
0
votes
0
answers
13
views
Do I have to use the 'one standard error from minimum CV error' rule? [duplicate]
I have just spent a decent amount of time learning how to do a stacked adaptive elastic net regression on several multiply-imputed datasets using the saenet package....
- 1,935
0
votes
0
answers
40
views
Quasibinomial GLMM with LASSO regularization in R
I am currently assessing drivers of deforestation using a GLM (generalised linear model) with LASSO regularization (using package glmnet in R). As the response variable is % of area deforested I have ...
3
votes
0
answers
39
views
How to show the existence of global minimizer of Lasso type of objective function?
Suppose the objective function to be minimized is
$$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$
where $\theta$ is the independent variable which is feasible in $\mathbb{R}^p$...
- 71
1
vote
0
answers
14
views
Increasing accuracy of prediction
I'm working with this data set trying to implement a model to predict the variable normexam. I've used the following models on sklearn, adding dummies for categorical variables, and got the following ...
- 113
2
votes
0
answers
54
views
Why do we need $\gamma>2$ in SCAD penalty?
The SCAD penalty $p(x | \lambda, \gamma)$ from https://myweb.uiowa.edu/pbreheny/7600/s16/notes/2-29.pdf or the paper "Variable Selection via Nonconcave Penalized Likelihood and its Oracle ...
- 71
1
vote
0
answers
10
views
Is it necessary to choose predictors for svm or I use all my variables?
I have transformed my categorical variables to dummies and I have used the lasso method to decide which variables I choose to do the logistic regression, my question is: for the svm model do I need to ...
- 13
1
vote
0
answers
11
views
MSE for ARIMA and UR in linear models
I'm trying to assess the predictive power of some time series models (LASSO, ARIMA, UR) on time series data, but I have a problem.
I am conducting simulations with $N$ data points, $P$ potential ...
- 11
0
votes
0
answers
7
views
Selecting sparse number of predictors each derived from a non-sparse b-spline in time
I have observations that I would like to model as responses at stations to slowly time-varying inputs at 80 candidate locations, few (perhaps 5-10?) of which are active and significant. I have ...
- 201
1
vote
0
answers
11
views
Can elasticnet ever select a different set of predictors than LASSO for a given lambda? [closed]
Since ridge regression can never penalize coefficients to zero, can elasticnet ever select a different set of predictors than LASSO for a given lambda?
- 2,076
0
votes
0
answers
28
views
Do we need to apply "multi-testing" corrections for the p-values in a regularized model?
Say we are fitting a penalized model, such as a linear regression with lasso regularization. We expect to obtain a model with the most significant covariables.
The method starts with many covariables ...
- 1,034
0
votes
0
answers
14
views
Lasso vs. Ridge Regression [duplicate]
The lasso coefficients are the ones that minimize $RSS+\lambda \Sigma_{j=1}^{p} |\beta_j|$ whereas the ridge regression coefficients those that minimize $RSS+\lambda \Sigma_{j=1}^{p} \beta_j^2$. I don'...
- 4,176
0
votes
1
answer
44
views
After selecting variables from lasso regression, is it a good practice to re-run the regression with selected variables?
As the subject suggests, after selecting regressors from lasso regression, is it a good practice to re-run the an ordinary linearly regression with selected variables?
I just feel like intuitively, ...
- 111
0
votes
0
answers
12
views
Graphical Lasso for estimating words network
I have a matrix whose columns are words and rows are different speeches by a person. Therefore, the i,j element of the matrix is the count of occurence of a word in a speech. I would like to estimate ...
- 143
1
vote
0
answers
42
views
Which evaluation metric should I choose? AIC or MSE?
I am currently at a total loss, so I hope someone can point me in the right direction regarding my model selection.
The situation
I want to create a linear model that best forecasts my data. I am ...
- 53
1
vote
1
answer
41
views
Tuning lambda in glmnet mgaussian multitask learning model for optimal support recovery
I was using a multivariate gaussian (mgaussian) glmnet model to solve the multitask learning problem below (deconvolution of a ...
- 3,021
1
vote
1
answer
40
views
Bootstrapped Prediction Interval for Adaptive Lasso
I am attempting to calculate a 95% prediction interval from an adaptive lasso model using the glmnet:: package in R. I adapted my method from the Python code in ...
- 111
0
votes
1
answer
32
views
From a computational perspective, how does the lasso regression shrink coefficients to 0?
I understand the analytic proof that lasso regularisation tends to shrink coefficients to zero.
However, from a practical standpoint, most of those methods are combined with gradient optimisation (...
1
vote
1
answer
106
views
Different sets of features selected by three different functions in R for running LASSO Regressions despite the same random seed for each
The GitHub Repository for this research project has all of the code included in this question.
Brief background context: I am just finishing up the work on my part as a coauthor on a research project ...
- 145
1
vote
0
answers
25
views
Fused lasso for image denonising
For a given data $y_{i}$, with $i=1, \dots, n$, we consider the following signal approximation:
$$
\hat{y} = \arg \min_{w}\sum_{i=1}^{n}(y_{i}-w_{i})^{2} + \lambda \sum_{(i,j)\in E}|w_{i} - w_{j}|,
$$
...
- 426
1
vote
0
answers
30
views
L1 vs L2 variance? [closed]
Which regularisation method L2 or L1 gives a lower variance?
$
f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta ||^2 \rightarrow L2
$
$
f(w) = \sum (\hat{y}_i - y_i)^2 + \sum || \beta || \rightarrow L1
...
user387816
1
vote
0
answers
72
views
Best Datasets and Packages for Comparing LASSO, Elastic Net, and Ridge
I have been recently been working with the MASS, lars, and glmnet packages to study variable ...
1
vote
0
answers
23
views
Differences in Performance Between in MASS Package lm.ridge() and enet in elasticnet Package
A background:
I am currently working with the 'elasticnet' package (elasticnet v.1.3) maintained by Hui Zou. This package was developed to accompany Hui Zou and Trevor Hastie's Statistical Society B ...
0
votes
0
answers
56
views
Relaxed Adaptive Lasso
I recently came across this study describing the benefits of the author's relaxed adaptive LASSO regression. The author describes a simple algorithm (which appears to be glmnet, effectively), as well ...
- 11
1
vote
0
answers
19
views
Simulation case for Double machine learning in Partial linear model with lasso where number of covariates is large
I want to conduct a simulation study on double machine learning in Partial linear regression setup as described in Chernozukov's paper for Double machine learning. I want to use lasso and want the ...
- 11
1
vote
0
answers
37
views
Penalty Factor for Adaptive Group LASSO
So gglasso in R appears to allow the user to implement adaptive group lasso by specifying group-wise penalty factors as you would with a typical adaptive lasso. Typically, when I want to train an ...
- 11
0
votes
0
answers
21
views
How is the intercept fit in glmnet when doing logistic regression?
In this question, it's explained how the intercept is fit under normal linear regression. It is given that it is calculated using
$$\beta_0 = \bar{y} - \sum_{j=1}^p \hat{\beta}_j \bar{x}_j$$
How is ...
- 105
1
vote
1
answer
154
views
Getting glmnet to select excatly given number of features?
I am using glmnet for feature selection, given a gaussian dependent variable. Part of my code is like this:
...
- 105
0
votes
0
answers
18
views
Is there any theory pertaining to an assumption to would allow the following inequality to hold (constrained LASSO)
Suppose we observe the vector-matrix pair $(y,\mathbf{X})\in\mathbb{R}^n\times\mathbb{R}^{n\times d} $ which is linked by the observation model:
\begin{equation}
y=\mathbf{X}\theta^*+\epsilon
\end{...
- 1,146
0
votes
0
answers
19
views
Should instrumental variables abide by the effect hierarchy principle?
People sometimes include the interactions between their instruments and region/year or the interactions between different instruments in the first stage of a 2SLS regression. I wonder if the effect ...
5
votes
0
answers
55
views
Generalize the 1SE rule to elastic net
When you do LASSO or ridge regression, and pick the hyperparameter using cross-validation, the 1SE rule suggest to select not the best CV result but the one with the most penalization that's still ...
- 29.3k
1
vote
1
answer
78
views
Interpreting survival analysis plot using genes as predictor
I will show what I'm doing it in R to make sure if I'm doing it correctly
This is my dataset which I'm using for analysis
...
- 155
4
votes
1
answer
217
views
Is standardization still needed after a LASSO model is fitted?
We know that it's better to standardization the training data (i.e. X_train) before fitting a LASSO model, especially when features are not in the same scale (Ref. Is standardisation before Lasso ...
- 143
0
votes
1
answer
35
views
Regression coefficient for downstream analysis from lasso regression
Fundamental question:
I have one dataset which I have used to build and generate a model for survival prediction.
Here I get like 40 genes as my predictor which I have tested both in my test and train ...
- 155
1
vote
0
answers
24
views
How to calculate (and conduct multiple test correction) for LASSO [duplicate]
I have a large dataset (1,465 observations, >50,000 predictors). I conducted a LASSO regression using glmnet in order to perform variable selection, then I ran a ...
- 11