Is a form of regularization used in estimation of regression coefficients which shrinks coefficient estimates by penalizing their absolute value (i.e. the $L_1$ norm of the estimates). The LASSO is equivalent to the Bayesian estimation problem where iid standard Laplacian prior is used for the ...
3
votes
1answer
108 views
Combining LASSO coefficients across imputed datasets
I am using the LASSO with multiple imputed datasets and I am not sure how should I combine the coefficients obtained on the different imputed datasets. I could simply average them (as I would do had I ...
0
votes
1answer
58 views
What does the index variable k define in the Lasso regularization function
In the Lasso L1 regularization, from where comes the value of the variable $k$ in the second part of the function? Why isn't it $n$, too?
$$L(\beta) = \sum_{i=1}^n (y_i - \phi(x_i)^T \cdot \beta)^2 + ...
1
vote
1answer
166 views
Model function for discovering irrelevant dimensions with L1 regularization
For homework I have been given a 20-dimensional input $x \in \mathbb{R}^{20}$, many of which are suspected to be irrelevant. I tried using L1-norm Lasso regularization to uncover which dimensions ...
3
votes
2answers
685 views
Why does Lasso do better than SVM?
This is a soft-question:
I have been evaluation various regression techniques over a regression dataset that I have. I am surprised by the fact that cross-validated RMSE of Lasso is better than SVM ...
2
votes
2answers
311 views
How to find parameters for ridge and lasso regularization when cost minimization does not converge?
In the Stanford ML course, we were taught to find good values for the lambda parameters of ridge/lasso by iterating for various lambda values on several cross-validation sets and picking the values ...
10
votes
3answers
4k views
How to estimate shrinkage parameter in Lasso or ridge regression?
I want to use Lasso or ridge regression for a model with more than 50,000 variables. I want do so using software package in R. How can I estimate shrinkage parameter ($\lambda$)?
Excuse me for the ...
7
votes
1answer
715 views
Is it possible to calculate AIC and BIC for lasso regression models?
Is is possible to calculate an AIC or BIC values for lasso regression models and other regularized models where parameters are only partially entering the equation. How does one determine the degrees ...
2
votes
1answer
448 views
Optimal parameter selection by repeated k-fold
I am working on Lasso problem and the selection of the optimal tuning parameter with $k$-fold procedure, say $k=10$.
Since this procedure relies on random subsampling, value of the optimal parameter ...
0
votes
1answer
742 views
Using glmnet to solve the LASSO problem
I have recently been made aware of the Lasso algorithm and found that the package
glmnet can be used to solve it. (I have the glmnet package on R).
If I have a matrix $A$ and a vector $y$ how do I ...
4
votes
1answer
160 views
Least angle regression for a set of vectors?
As far as I know, LARS solves the following problem (using the same notation as Efron et al. Least angle regression):
Given a vector y, and a matrix X. Pick some column vectors from X, and express ...
3
votes
4answers
325 views
Bayesian prior corresponding to penalized regression coefficients
I'm working on a Bayesian Regression problem where I would like to estimate the beta coefficients subject to this constraint (penalty):
$\sum|\beta_i|<C$ or similarly $\sum \beta_i^2<C$
Which ...
12
votes
1answer
255 views
LASSO/LARS vs general to specific (GETS) method
I have been wondering, why are LASSO and LARS model selection methods so popular even though they are basically just variations of step-wise forward selection (and thus suffer from path dependency)?
...
1
vote
0answers
100 views
Cox model with Lasso for PH test and P value
I am using the penalized package in r for fitting a Cox model with a lasso penalty.
Does the Cox model + lasso need to test proportionality?
How do you get the ...
5
votes
1answer
599 views
Can compressed sensing be applied to data mining?
For high dimensional data, LASSO is useful as it allows to determine the few significant covariates. I think compressed sensing is an upgrade of LASSO. According to the wiki page:
[the LASSO ...
19
votes
4answers
947 views
What problem do shrinkage methods solve?
The holiday season has given me the opportunity to curl up next to the fire with The Elements of Statistical Learning. Coming from a (frequentist) econometrics perspective, I'm having trouble grasping ...
3
votes
1answer
600 views
Need for centering and standardizing data in regression
Consider linear regression with some regularization:
E.g. Find $x$ that minimizes $||Ax - b||^2+\lambda||x||_1$
Usually, columns of A are standardized to have zero mean and unit norm, while $b$ is ...
11
votes
1answer
286 views
Lasso modification for LARS
I am trying to understand how Lars algorithm can be modified to generate Lasso. While I do understand LARS, I am not able to see the Lasso modification from the paper by Tibshirani et al. In ...
4
votes
1answer
972 views
Derivation of closed form lasso solution
For the lasso problem
$\min_\beta (Y-X\beta)^T(Y-X\beta)$ such that $\|\beta\|_1 \leq t$. I often see the soft-thresholding result
$$ \beta_j^{\text{lasso}}= ...
16
votes
2answers
3k views
What is the lasso in regression analysis?
I'm looking for a non-technical definition of the lasso and what it is used for.
0
votes
1answer
51 views
Using monomvn to impute monotone missings
I am attempting to use the monomvn package to impute monotone missings. I've followed the basic example:
data(cement.miss)
out <- monomvn(cement.miss)
...
5
votes
2answers
936 views
How is the intercept computed in GLMnet?
I've been implementing the GLMNET version of elastic net for linear regression with another software than R. I compared my results with the R function glmnet in lasso mode on diabetes data.
The ...
12
votes
1answer
403 views
Reproducing table 18.1 from “Elements of Statistical Learning”
Table 18.1 in the Elements of Statistical Learning summarizes the performance of several classifiers on a 14 class data set. I am comparing a new algorithm with the lasso and elastic net for such ...
3
votes
2answers
175 views
The effectiveness of coordinate ascent
Why is it so successful for the lasso, though for most other problems standard Quasi-Newton approaches seem to be preferred? I sort of have this vague geometric idea that it might have to do with the ...
7
votes
2answers
444 views
Optimal penalty selection for lasso
Are there any analytical results or experimental papers regarding the optimal choice of the coefficient of the $\ell_1$ penalty term. By optimal, I mean a parameter that maximizes the probability of ...
12
votes
2answers
535 views
Fitting an ARIMAX model with regularization or penalization (e.g. with the lasso, elastic net, or ridge regression)
I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a ...
0
votes
1answer
262 views
Software for LASSO for high dimensional dataset
I have a dataset with between 10,000 and 100,000 feature values. The number of datapoints is between 1,000 and 10,000. I want to perform a LASSO on this dataset but can't really find any good software ...
4
votes
1answer
2k views
How to perform logistic regression with lasso using GLMSELECT?
I am trying to perform logistic regression with lasso. For the logistic regression part I am using PROC LOGISTIC but I am not sure how to do lasso with ...
4
votes
1answer
680 views
Cox model with LASSO
Rob Tibshirani propose to use lasso with Cox
regression for variable selection in his 1997 paper
"The lasso method for variable selection in the Cox
model" published in Statistics In Medicine 16:385.
...
4
votes
1answer
149 views
Regularized fit from summarized data
I have a multiple linear regression problem $y=X\beta+\epsilon$. The number of observations $m$ is large, so by the time the data gets to me it's been summarized into:
$m$
$X^TX$
$X^Ty$
$y^Ty$
...
14
votes
3answers
2k views
What are disadvantages of using the lasso for variable selection for regression?
From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
9
votes
1answer
2k views
Estimating R-squared and statistical significance from penalized regression model
I am using the R package penalized to obtain shrunken estimates of coefficients for a dataset where I have lots of predictors and little knowledge of which ones are important. After I've picked tuning ...
7
votes
4answers
3k views
GLMNET or LARS for computing LASSO solutions?
I would like to get the coefficients for the LASSO problem
$$||Y-X\beta||+\lambda ||\beta||_1.$$
The problem is that glmnet and lars functions give different answers. For the glmnet function I ask ...
8
votes
1answer
577 views
LASSO assumptions
In a LASSO regression scenario where
$y= X \beta + \epsilon$,
and the LASSO estimates are given by the following optimization problem
$ \min_\beta ||y - X \beta|| + \tau||\beta||_1$
Are there any ...
4
votes
2answers
321 views
Stochastic coordinate descent for $\ell_1$ regularization
I recently came across the following paper: "Stochastic Methods for $\ell_1$ Regularized Loss Minimization" by Shai Shalev-Shwartz and Ambuj Tewari, ICML 2009.
In the paper, the authors propose a ...
7
votes
3answers
904 views
Lasso fitting by coordinate descent: open-source implementations?
What open-source implementations -- in any language -- exist out there that can compute lasso regularisation paths for linear regression by coordinate descent?
So far I am aware of:
glmnet
...
6
votes
1answer
671 views
LARS vs coordinate descent for the lasso
What are the pros and cons of using LARS [1] versus using coordinate descent for fitting L1-regularized linear regression?
I am mainly interested in performance aspects (my problems tend to have ...
1
vote
2answers
589 views
Coordinate descent for the lasso or elastic net
Are there any good papers or books dealing with the use of coordinate descent for L1 (lasso) and/or elastic net regularization for linear regression problems?
15
votes
4answers
2k views
Least-angle regression vs. lasso
Least-angle regression and the lasso tend to produce very similar regularization paths (identical except when a coefficient crosses zero.)
They both can be efficiently fit by virtually identical ...
4
votes
1answer
193 views
Updating the lasso fit with new observations
I am fitting an L1-regularized linear regression to a very large dataset (with n>>p.) The variables are known in advance, but the observations arrive in small chunks. I would like to maintain the ...
4
votes
3answers
849 views
Java implementations of the lasso
Are there any open-source Java implementations of lasso or least angles regression?
Pure Java code would be best, but clean implementations in other languages would also be of interest. I am already ...
6
votes
1answer
1k views
In R, does “glmnet” fit an intercept?
I am fitting a linear model in R using glmnet. The original (non-regularized) model was fitted using lm and did not have a ...
22
votes
2answers
2k views
When to use regularization methods for regression?
In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS?
In case this helps steer the discussion, my main interest is improving ...
8
votes
3answers
2k views
How to apply LASSO to IRLS (logistic regression)?
I have programmed a logistic regression using the IRLS algorithm. I would like to apply a LASSO penalization in order to automatically select the right features. At each iteration, the following is ...
8
votes
2answers
1k views
Soft-thresholding vs. Lasso penalization
I am trying to summarize what I understood so far in penalized multivariate analysis with high-dimensional data sets, and I still struggle through getting a proper definition of soft-thresholding vs. ...
29
votes
3answers
6k views
When should I use lasso vs ridge?
Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
