# Tagged Questions

A regularization method for regression models that shrinks coefficients towards zero, making some of them equal to zero. Thus lasso performs feature selection.

6 views

### Bias of the Lasso estimator from Tibshirani

I am searching for a Theorem that gives upper bounds for the bias of the Lasso estimator from Tibshirani. Do anybody know such a Theorem? Thanks a lot for help! Best regards, Markus
48 views

### What are the pros and cons of employing LASSO for causality analysis?

It looks like social sciences are impressed by Statistical Learning and its results. A couple of months ago, I heard Imbens saying: "LASSO is the new OLS". My problem with this is that I've been ...
17 views

### Select and weigh questions on a reading assessment

I am a phd student in computer science and as such I am the goto guy for anything "mathy" in the cross discipline research group to which I belong. I have recently been given an assignment at work I'm ...
38 views

### LASSO regression when model is known

I am very new to regression as I have been reading "The Elements of Statistical Learning: Data Mining, Inference, and Prediction" by Hastie et al. on Standford's website this weekend. My goal is to ...
50 views

### R LASSO always include some coefficient and question about data partition

I have limited statistic knowledge but I am trying to conduct logistic regression by using a data with 300+ predictors. So I decided to use glmnet and LASSO. Below please see my code: ...
29 views

### Are LASSO coefficients raw or standardized? [on hold]

I'm doing binary classification and ran LASSO to try and do feature selection to reduce the parameters in the model. I have the coefficients from glmnet at the ...
44 views

### Optimization with both L1 and L2 regularization

After doing some research I suppose the hard part is that, L2 regularized problem is often solved by gradient descent, while L1 regularized problem is often solved by coordinate descent. But which ...
36 views

### R - glasso very slow for high feature space

all, I'm doing a graphical lasso in order to approximate the inverse of the covariance matrix of a 1200 (p-features) by 100 or so (n observations) data matrix. Basically, I'm inverting a 1200 x 1200 ...
64 views

### Cross validated penalized logistic regression - one standard deviation rule

I am new to this topic and would like to understand it better. I want to build a binary classifier based on penalized logistic regression. I have 10 features and 23 observations: 16 from class "0" and ...
9 views

### How to use adaptive lasso to do subset selection with longitudinal data?

How to use adaptive lasso to do subset selection with longitudinal data? Use R packages Adaptive lasso in R In the website above introduces the R code,but I don't know how to use it with longitudinal ...
36 views

### R logistic regression optimal cut point

I am working on a dataset that has 300+ predictors and the dependent variables is very imbalanced (99:1). I need to have a prediction accuracy to show to my client.Here is my analytical process. ...
36 views

### How to imagine (visualize) the difference between LARS and Lasso

I'm reading the LARS paper. It turns out the solution path of LARS is quite similar with Lasso, and that paper has an explanation in section 3.1. An important fact ...
96 views

### Lasso Regression for predicting Continuous Variable + Variable Selection?

I'm attempting to predict vegetation productivity based on climatic and land use variables (the latter are categorical). I found that there is a multicollinearity problem between the predictors ...
40 views

### Convergence analysis for forward stagewise regression?

Forward stagewise regression is a simple model selection algorithm related to least angle regression and LASSO. (see e.g. the LARS paper) It repeats the following steps, initializing a predictor ...
34 views

### Why do I get worse regression metrics when I add more instances to the problem?

I find this counter-intuitive. First I chose randomly 7000 instances and my model explains 55% of the variance. Then I train with the whole dataset (43000) and I get negative $R^2$. How is this even ...
116 views

### If the LASSO is equivalent to linear regression with a Laplace prior how can there be mass on sets with components at zero?

We are all familiar with the notion, well documented in the literature, that LASSO optimization (for sake of simplicity confine attention here to the case of linear regression)  {\rm loss} = || y - ...
33 views

### Meaning and significance testing of coefficients in lasso/ridge regression

Can somebody explain the importance of significance testing in ridge/lasso regression? Is it necessary to do it? And, how can we interpret the coefficients of ridge regression which are penalized?
21 views

### Unit tests for lasso?

I have my own implementation for non-negative lasso, but not sure if it is right. Any idea where can I find unit tests to verify my code?
10 views

### Lasso - standardize y or only x? [duplicate]

In lasso, I am standardizing the input, x. What do I need to do with the output, y? Are any transformations necessary? Only centering? Both centering and scaling? This question is not a duplicate ...
33 views

### Lasso variable selection through correlation?

In lasso, we first standardize all variables to mean=0, var=1. As such, a beta is simply correlation, right? We want to keep the sum of absolute values of betas below certain value, so what do we do: ...
64 views

### Elastic net produces complex output with too many non-zero coefficients

I have run 3-fold cross-validation for elastic net using elasticnet R function on ~200 observations and using 80 variables (and there will be some more). Both ...
27 views

### Adaptive Group Lasso

Is there any code/algorithm (preferrably for MatLab, but R is fine) where I can easily implement the Adapative Group Lasso as in here. I have found algorithms to implement the group lasso with a ...
6 views

### How do I compare the influence of variables against each other?

I've made 22 species specific multiple linear regression models using LASSO and would like to see which variables have the greatest impacts among the models. I'd like to use the parameter values to ...
28 views

### How does glmnet() handle with both penalized and unpenalized covariates?

Is it possible to do a lasso model with both penalized and un-penalized covariates? That is, I want to do an estimate with Y ~ gamma * X + beta * Z, where X is a ...
7 views

### How to see the adjR-square in Lasso Regression?

After doing lasso, the final parameters are only 6, but I have 200 covariates originally, is it too literally? And how to see the correspond adj R-square in Lasso Regression? By the way, I also tried ...
60 views

### Mixed effects Lasso model setup in R, for high dimensional data

My goal is to model the relationship between RETURN and SCORE from my survey dataset with the following structure: RETURN (numeric continuous) = company share price performance SCORE (numeric ...
66 views

### Help with modeling an insurance data set with LASSO regression in R

I'm hoping a few folks will help me conceptually with building a model based on an insurance dataset. I'm using LASSO for its feature selection, and I'm using R - probably either the glmnet or ...
58 views

### Does logistic regression with LASSO require smaller samples to train in comparison with regular logistic regression?

The question is within the title.
50 views

### Interpretation of elastic net coefficients under multicollinearity

I am studying the elastic net regression and in some material I read, it was mentioned that the method will choose a group of regressors that are correlated while LASSO can pick one among the ...
369 views

### What is elastic net regularization, and how does it solve the drawbacks of Ridge (L2) and Lasso (L1)?

Is elastic net regularization always preferred to Lasso & Ridge since it seems to solve the drawbacks of these methods? What is the intuition and what is the math behind elastic net?
504 views

### When will L1 regularization work better than L2 and vice versa?

Note: I know that L1 has feature selection property. I am trying to understand which one to choose when feature selection is completely irrelevant. How to decide which regularization (L1 or L2) to ...
10 views

### How to model the discrepancy between test result and prediction code, and find the main sources of discrepancy

I have data about some tested machines, stored as a database with $\approx$ 300 observations, 40 predictors, 2 outcomes. The two outcomes (responses) are the result of the machine test for that ...
20 views

### Intepreting significance values for the LASSO regression with covTest in R

The covTest package in R gives significance values for a LASSO regression. How should the results be interpreted? I get negative predictor numbers and NAs for the p-values. What do these mean? More ...
41 views

### lasso - how to evaluate results

I'm working on lasso as an alternative to step-wise forward/backward regression using the lars package in R. I normalized my variables, calculated the ...
180 views

### Why is Lasso penalty equivalent to the double exponential (Laplace) prior?

I have read in a number of references that the Lasso estimate for the regression parameter vector $B$ is equivalent to the posterior mode of $B$ in which the prior distribution for each $B_i$ is a ...
91 views

### Recommend a method for variable selection (other than classification tree or random forest)?

Just wonder if you could recommend a few methods (other than tree-based methods) to analyze a dataset in which n= 350 and p = 35. The goal is not so much about prediction, but to find/select ...
19 views

### Feature selection for linear regression using bootstrapped RMSE as criteria

I'm trying to build robust linear regression model (lmrob from robustbase) using several (< 15) features. I know that traditional stepwise algorithms aren't the best alternative since they are ...
48 views

### True Test Error for LASSO

I have a data set which is split into a training set and a test set. Half is training and half is test. I apply OLS using lm in R, 10 fold Cross Validated LASSO and 10 fold Cross Validated RIDGE using ...
31 views

### Conditional Density for Sigma (Bayesian Lasso)

I found that in Bayesian Lasso commonly $\beta \sim N(0,\sigma^2*diag(\tau))$ and $\sigma,\tau \sim \pi(\sigma,\tau)$ is used. Whereas $\pi(\cdot)$ is a product of Laplace distributions. Is it ...
111 views

### Scikit's prediction for linear model

Looking at this example for the Lasso method: ...
39 views

### GLMNET small deviance explained; reuse of selected predictors in other model

I am trying to run glmnet for logistic regression (I have some continuos predictors which I have scaled with scale() and some categorical which I turned to dummy predictors, 27 predictors, 800 ...
1k views

### Why does shrinkage work?

In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive ...
169 views

### Why can't ridge regression provide better interpretability than LASSO?

I already have an idea about pros and cons of ridge regression and the LASSO. For the LASSO, L1 penalty term will yield a sparse coefficient vector, which can be viewed as a feature selection method. ...
63 views

### Why are all Lasso coefficients in model 0.0?

I'm using from sklearn.linear_model import Lasso in Python 2.7.6 I wrote a script that I've used for doing a Lasso regression for my Features (X) and my Targets ...
44 views

### Ordinary linear regression vs penalized regression: predictive MSE

Does predicting on a test set with an ordinary linear regression model result in a smaller predictive MSE compared to a penalized regression model (LASSO or ridge)?
43 views

### Subset selection, shrinkage and dimensionality reduction in regression analysis

I am currently reading An Introduction to Statistical Learning: With Applications in R by Robert Tibshirani and Trevor Hastie. I am confused about various regularization methods for linear regression, ...
46 views

### Generating Lasso Path for Feature Selection

I am building a Logistic regression model and exploring LASSO for feature selection. I generate the lasso path using the following code: ...
30 views

### Why lasso regression is equivalent to finding the least loss function such that $\sum_{j=1}^{p} |\beta_j| \leq \lambda$? [duplicate]

Why is minimizing $\frac{1}{2N} \sum_{i=1}^{N} (y_i - \beta_0 - x_i^T \beta)^2 + \lambda \sum_{j=1}^{p} |\beta_j|$ equivalent to finding the minimum of \$\frac{1}{2N} \sum_{i=1}^{N} (y_i - \beta_0 - ...