Skip to main content

Questions tagged [lasso]

A regularization method for regression models that shrinks coefficients towards zero, making some of them equal to zero. Thus lasso performs feature selection.

Filter by
Sorted by
Tagged with
0 votes
1 answer
12 views

Reducing Variance with Regularization in LOOCV for Small Datasets

I have a small dataset and I am considering using Leave-One-Out Cross-Validation (LOOCV) to evaluate my model. I understand that cross-validation, in general, is a method to assess a model's ...
oriKAN's user avatar
  • 1
0 votes
0 answers
39 views

How to determine lambda for graphical lasso?

I am trying to figure out how to determine lambda for a graphical lasso. I have found that someone had the exact same question that me 9 years ago. I was wondering if anything exists in R to determine ...
Simon's user avatar
  • 1
1 vote
0 answers
11 views

Assessing Random Search Cross Validation: Tuning in ElasticNet with Large Feature Sets

I'm working on estimating an ElasticNet model for a large dataframe with over 100,000 variables, resulting in a well overidentified scenario. To tune my model, I've set up a grid of hyperparameters (...
george1994's user avatar
0 votes
1 answer
47 views

why does LASSO regression return unstandardized coefficients [closed]

I have more general questions that does not refer to a coding issue. Why does LASSO regression require standardization of the predictors but return unstandardized coefficients (glmnet function - https:...
Simon's user avatar
  • 1
1 vote
0 answers
89 views

AUC > 0.5 under null model following feature selection

I've been going over the output of a Monte Carlo model that simulates disease risk as a function of genotype. Under a null model of no disease risk, we have 1000 case and 1000 control individuals. ...
Max's user avatar
  • 133
1 vote
0 answers
14 views

penalized package [closed]

Has anyone used penalized package? I was using it for lasso in Cox regression, with time-varying coefficients. The problem is when I made a plot with ...
Danny's user avatar
  • 11
4 votes
1 answer
45 views

The sum of $O_p$ --$ O_p \left(s^2\frac{\log d}{n}+s\sqrt{\frac{\log d}{n}} \right) $

I read papers in the area of inference for high-dimensional graphical models and these papers always state the convergence rate of the estimator. Using $O_p$ is a good choice. Maybe I made some ...
mathhahaha's user avatar
3 votes
1 answer
38 views

What is the boundary curve for $λ_1$ and $λ_2$ that give at least a 0 component in elastic net?

Define the elastic net estimate: $ \hat{\beta}^{\lambda_1, \lambda_2} = \arg \min_{\beta \in \mathbb{R}^p} \left( \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda_1 \ \frac{1}{2} \|\beta \|_2^2 + \lambda_2 ...
george1994's user avatar
1 vote
1 answer
25 views

When does a extended BIC curve for a Gaussian Graphical model/GLasso look incorrect?

I have a model for a network, and I wanted to analyze the extended BIC curve for a graphical lasso model as according to Foygel and Drton 2010. The paper gives a list of assumptions for the data/model ...
Robertmg's user avatar
  • 121
1 vote
0 answers
29 views

Unacceptable results for adj R2

I have a dataset with 19 features. When I ran it with the Lasso algorithm. R2 for test and train was 0.69. But the value of adj r2 for test is 1.28 (above 1), and for train the value is 0.28. What is ...
Erfan Mollai's user avatar
4 votes
1 answer
176 views

Motivation for automated variable selection in case of p>n

I have written the following text as a motivation for using automated variable selection in cases where the number of variables (p) is greater than the number of observations (n). However, I am not ...
george1994's user avatar
0 votes
0 answers
38 views

Evaluating Lasso's Unique Solution and its consequences in applications?

I've grasped from a paper (https://www.stat.cmu.edu/%7Eryantibs/papers/lassounique.pdf) that Lasso may not yield a unique solution when the number of variables (p) exceeds the number of observations (...
george1994's user avatar
0 votes
0 answers
48 views

Behavior of Lasso Estimator with More Predictors Than Observations (p > n) and Identical Correlations?

What is the behavior of a Lasso estimator if it is used in a dataset with more predictors (p) than observations (n), where all predictors are uncorrelated but highly relevant to 𝑦 y with exactly the ...
george1994's user avatar
1 vote
0 answers
31 views

Constructing subgroup comparisons for variable selection

I am trying to construct a covariate according to the the following description: When constructing the covariates to measure a subgroup effect, we generate covariates that capture the causal effect ...
T-Porsch's user avatar
-1 votes
2 answers
42 views

Does penalizing the slope in Ridge/ Lasso regression has adverse effect based on the training data?

I have just started to learn ridge and lasso regression (by this YouTube video). To my understanding, these regressions are similar to linear regression, but we penalize the higher slope by the ...
Soheil's user avatar
  • 121
0 votes
0 answers
34 views

Is there any test I can apply to the data to tell whether the adaptive LASSO or the LASSO is likely to perform better in prediction?

Is there a. test I can perform on a sample that will tell me if coefficients estimated using the LASSO, the adaptive LASSO, or the relaxed adaptive LASSO are likely to give better (in the mean squared ...
andrewH's user avatar
  • 3,157
4 votes
2 answers
55 views

Is lasso preferable to ridge or principal component regression in multicollinear settings?

Consider a $N\times p$ data matrix $\mathbf X$ with columns $\mathbf x_j$. ESL recommends standardizing the inputs before performing ridge regression, which I understand to mean centering the columns $...
Tomo's user avatar
  • 141
4 votes
2 answers
89 views

Finding the corners of noisy polygons

I have some polygons that look for example like this: If I zoom in very close on one side, you can see the noise. The data is a list of x coordinates and a corresponding list of y coordinates. I ...
sav's user avatar
  • 239
0 votes
0 answers
42 views

Question about LASSO, RMSE, and Standardization

I have a question about doing LASSO in R using glmnet. It's kind of a conceptual question; I learned that we should interpret RMSE after performing ...
noone's user avatar
  • 43
2 votes
0 answers
33 views

Regression with small sample size - LASSO or remove variables?

I'm trying to run a regression, but I only have 14 observations, each being a different city in the US. My dependent variable is the total number of trips per capita, and my explanatory variables are ...
BeyondConfused's user avatar
3 votes
0 answers
41 views

L1/L2 regularization in neural nets

For linear regression, after doing L1/L2 regularization one can compute a closed form solution for the weights in nice cases. From here, one gets the intuition where: L2 regularization shrinks ...
dummy's user avatar
  • 275
0 votes
0 answers
8 views

Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?

I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
CodusOProgrammatus's user avatar
2 votes
0 answers
19 views

Implementing a SEM with lasso regression [closed]

I have the following structural equation model (SEM) structure in mind: In reality there are more than 3,2, resp. 3 variables, so I want to use lasso regression for the arrows. Is it possible to ...
Hendrik's user avatar
  • 21
0 votes
0 answers
17 views

Target variable standardization for lasso regression

I am working with different models for a regression task. The range of my target variable is very small: I noticed a very bad performance of the lasso regression and elastic net model in comparison ...
Limmi's user avatar
  • 13
4 votes
1 answer
177 views

Stochastic boundedness in consistency proof

I'm reading Knight and Fu (2000), Asymptotics for Lasso-Type Estimators and I don't understand why (6) and (7) imply consistency in Theorem 1 (copied and pasted below). I'm familiar with the standard ...
Giacomo's user avatar
  • 163
1 vote
0 answers
40 views

Knockoff R package removing ALL variables based on LASSO, Why?

I am simulating a data set from a linear regression model and selecting the variables using LASSO (glmnet). The selection works relatively well with ...
Jack's user avatar
  • 11
6 votes
1 answer
182 views

How to adjust p-values in a multiple regression when variables have been a selected by Lasso?

I have designed an analysis where I am testing a lot of variables together. So I first apply a Lasso regression to select the top variables, and then I run a standard (unregularized) multiple ...
PlasmaBinturong's user avatar
0 votes
0 answers
22 views

Lasso Regression Problem [duplicate]

$\operatorname*{argmin}_\beta\{\|y-X\beta\|^2 + \lambda\|\beta\|_1$, where $X$ is orthonormal. $\beta \in \mathbb R^d$. $X = [x_1,\ldots,x_n]^T$ and $y=(y_1,\ldots,y_n)^T \in \mathbb R^n$. $X^TX=I_{d\...
Harry Lofi's user avatar
1 vote
1 answer
64 views

What does it mean for a variable to be statistically significant but not selected as important by classification methods?

I'm basically playing around with some lipidomics data to practice, so my question is purely theoretical. I wanted to see if I could find lipid classes that differ between two groups and I was ...
maglorismyspiritanimal's user avatar
0 votes
0 answers
31 views

Derivation of bias of LASSO in the ortnormal case

In the following lecture slides by Breheny, P. (2016) titled "Adaptive lasso, MCP, and SCAD" from the High Dimensional Data Analysis course at the University of Iowa, slide 2 presents the ...
Joe94's user avatar
  • 95
0 votes
0 answers
33 views

Sample Size for Adaptive Lasso

Be gentle, I'm learning here. I have a fairly simple adaptive lasso regression that I'm trying to test for a minimum sample size. I used cross-validated mean squared error as the "score" of ...
JRW's user avatar
  • 1
0 votes
0 answers
17 views

Lasso regression test MSE lower than train MSE

Im currently using Lasso to build a predictive model for numeric variable . Before scaling the features I split the data for train test and validation . I have a feature named 'year' and i wanted the ...
liza read's user avatar
1 vote
0 answers
32 views

The importance of stationarity for the oracle property of Elastic Net Regression?

I've been on the lookout for a while, but unfortunately, I'm still coming up empty-handed in my search for papers or books that dive into the theoretical derivation or simulation of the impact of non-...
Joe94's user avatar
  • 95
0 votes
0 answers
19 views

How to sample with the 1-norm?

I am currently working on ridge regression, which can be interpreted using Bayesian statistics (DOI: 10.1016/j.electacta.2015.03.123). In particular, I know that the maximum-a-posteriori (MAP) ...
user avatar
0 votes
0 answers
19 views

How can I implement a linear mixed model on data where the reference doesn't have repeated measures?

I want to analyze in paired data that has many dependent variables (continuous) and several categorical predictors. The predictors are: Group (patient/control), Treatment (none, t1, t2, t3), and Gene (...
maglorismyspiritanimal's user avatar
1 vote
0 answers
58 views

Sensitivity analysis of features in a Random Forest Model

I have a built a large Random Forest Classifier and was able to output the feature importance as below: I understand that this importance is a based on mean decreased impurity. But how to interpret ...
Cathy's user avatar
  • 11
2 votes
1 answer
64 views

Double selection lasso in and NA's handling

I work in a team where everyone uses Stata, and I work in R. I have created an efficient workflow that allows me to export the results quickly. The problem I ran into was when implementing the double ...
Paula's user avatar
  • 151
1 vote
0 answers
19 views

What to do if the standard error estimates for adaptive Lasso estimates are useless?

This is more of an advisory question. I am researching the performance of the adaptive Lasso on multidimensional GARCH MIDAS data under regime switching and other confounders for my bachelors. I know ...
mexx's user avatar
  • 23
0 votes
0 answers
21 views

Deriving a design Matrix for penalized regression [duplicate]

I am having issues attempting to derive this new design matrix. The objective function for the previous question was as follows: $\sum_{i}^{n}(Y_{i}-\mu)^2+\lambda\mu^2$ Find a design matrix $X(\...
Harry Lofi's user avatar
0 votes
1 answer
60 views

Obtaining P value in LASSO regularized linear regression showing that the model is generalizable

I have a problem. I am a biologist working in machine learning. Recently I am dealing with LASSO regularized linear regression. Very nice RMSE, MSE, R^2 values under all kinds of cross-validation. But ...
Mátyás Bukva's user avatar
3 votes
2 answers
129 views

How many zeroes from lasso linear regression?

Given a dataset $X$ with $d$-dimensional features $x \in R^d$, and a response variable $y$ you can perform a lasso regression, ie linear regression with L1 regularization, as $$ \min_{\beta} (X\beta - ...
alexmolas's user avatar
  • 288
0 votes
0 answers
37 views

Restricting the order in which variables are allowed to enter in the LARS algorithm

In the Least Angle Regression paper, in section 3, the authors refer to how you can restrict the order in which variables are allowed to enter the LARS Algorithm. In particular, having obtained some ...
hahnbanach123's user avatar
8 votes
1 answer
415 views

Why not directly brute force sparsify the OLS estimator instead of using Lasso?

I have a question about the Lasso estimator. I understand that it is particularly useful in high-dimensional settings due to its sparsity-inducing properties. For instance, if the design matrix is ...
user405777's user avatar
2 votes
1 answer
147 views

What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
AL1117's user avatar
  • 31
2 votes
0 answers
30 views

Transforming discrete optimisation problem into continuous optimisation problem

In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
LoveRKHS's user avatar
5 votes
1 answer
111 views

Does performing Likelihood Ratio Test to compare two nested LASSO models make statistical sense?

From what I've studied, the LRT is used to compare two nested models, i.e. 2 models having different sets of nested features, in my case e.g. Model1: binary_outcome ~ X1 + X2 Model2: binary_outcome ~ ...
Argh__1's user avatar
  • 53
1 vote
0 answers
33 views

Group Lasso optimization

I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \...
Jenny's user avatar
  • 261
1 vote
0 answers
31 views

LASSO number of coefs reduces with features increase

There seems to be a reverse correlation to the number of coefs I end up with, to the number of features. I.e. as I add features, I end up with less non-zero coefficients. Seem counter-intuitive to me. ...
ManInMoon's user avatar
  • 223
1 vote
0 answers
42 views

Handling Mixed-Frequency time series data for Feature Selection

I'm currently working on a project where I aim to apply LASSO regularization and conduct variable importance analysis on WTI crude oil prices. My challenge is dealing with datasets that have different ...
Dome's user avatar
  • 21
1 vote
0 answers
42 views

Lasso model by glmnet package in small sample size [closed]

I want to create a lasso model, but I can't have a test set because I have a small sample size. So, I want to use cross-validation to evaluate the model. I have seen that functions like ...
muhammad's user avatar

1
2 3 4 5
30