35
votes
Accepted
Why does glmnet use "naive" elastic net from the Zou & Hastie original paper?
I emailed this question to Zou and to Hastie and got the following reply from Hastie (I hope he wouldn't mind me quoting it here):
I think in Zou et al we were worried about the additional bias, but ...
- 107k
34
votes
Why is glmnet ridge regression giving me a different answer than manual calculation?
The difference you are observing is due to the additional division by the number of observations, N, that GLMNET uses in their objective function and implicit standardization of Y by its sample ...
- 511
23
votes
Does caret train function for glmnet cross-validate for both alpha and lambda?
Old question, but I recently had to deal with this problem and found this question as a reference.
Here is an alternative approach:
The glmnet vignette (https://web.stanford.edu/~hastie/glmnet/...
- 491
22
votes
Accepted
Caret glmnet vs cv.glmnet
I see two issue here. First, your training set is too small relative to your testing set. Normally, we would want a training set that is at least comparable in size to the testing set. Another note is ...
- 336
22
votes
How to obtain Confidence Intervals for a LASSO regression?
Please think very carefully about why you want confidence intervals for the LASSO coefficients and how you will interpret them. This is not an easy problem.
The predictors chosen by LASSO (as for any ...
- 102k
21
votes
LASSO with interaction terms - is it okay if main effects are shrunk to zero?
I am late for a party, but here are few of my thoughts about your problem.
lasso selects what is informative. Lets consider lasso as a method to get the highest predictive performance with the ...
- 7,779
20
votes
Accepted
How to interpret Lasso shrinking all coefficients to 0?
I don't think you have made a mistake in the code. This is a matter of interpreting the output.
The Lasso doesn't indicate which individual regressors are "more predictive" than others. It simply ...
- 334k
19
votes
Accepted
LASSO with interaction terms - is it okay if main effects are shrunk to zero?
One difficulty in answering this question is that it's hard to reconcile LASSO with the idea of a "true" model in most real-world applications, which typically have non-negligible correlations among ...
- 102k
17
votes
Accepted
Why are confidence intervals and p-values not reported as default for penalized regression coefficients
Little late to the party, but in case anyone stumbles across this question in the future. . . .
Best answer: have a look at section 6 of the vignette for the penalized R package ("L1 and L2 ...
- 753
17
votes
Is R's glm function useless in a big data / machine learning setting?
This has nothing to do with glm, you simply created a problem with an artificial perfect separation:
...
- 36.3k
16
votes
Accepted
Choosing optimal alpha in elastic net logistic regression
Clarifying what is meant by $\alpha$ and Elastic Net parameters
Different terminology and parameters are used by different packages, but the meaning is generally the same:
The R package Glmnet uses ...
- 9,986
16
votes
Accepted
Is R's glm function useless in a big data / machine learning setting?
The unregularized model is suffering from complete separation because you are trying to predict the dichotomized variable price_c from the continuous variable ...
- 12.9k
12
votes
Accepted
Replicating results for glmnet linear regression using a generic optimizer
tl;dr version:
The objective implicitly contains a scaling factor $\hat{s} = sd(y)$, where $sd(y)$ is the sample standard deviation.
Longer version
If you read the fine print of the glmnet ...
- 3,004
12
votes
Tune alpha and lambda parameters of elastic nets in an optimal way
Cross-validation is a noisy process and you shouldn't expect the results from two runs to be similar, even if everything is working fine. You can try repeating your experiment several times and see ...
- 8,349
11
votes
Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression?
Breiman et al.'s book (cited in the other answer's quote from Krstajic) is the oldest reference I've found for the 1SE rule.
This is Breiman, Friedman, Stone, and Olshen's Classification and ...
- 4,603
11
votes
Lasso vs. adaptive Lasso
Adaptive LASSO is used for consistent variable selection. The problems we encounter when using the LASSO for variable selection are:
The shrinkage parameter must be larger for selection than ...
- 720
11
votes
Accepted
Is the LASSO really applicable for binary classification problems?
It is valid. Note the family="binomial" argument which is appropriate for a classification problem. A normal lasso regression problem would use the gaussian link ...
- 2,842
11
votes
Accepted
ridge and lasso models in caret with lambda=0
I think that somewhat unfortunately you have hit a minor bug in caret's implementation of the glmnet model. (Bug in the sense of ...
- 46.1k
10
votes
Accepted
glmnet - compute maximal lambda value
The smallest value of lambda for which no parameters are selected may be computed by
$\max_j \frac{1}{\alpha n} \sum_{i=1}^n [Y_i - \bar Y (1- \bar Y)] X_{ij}$
See my example:
...
- 170
10
votes
How to interpret coefficients of a multinomial elastic net (glmnet) regression
I emailed kind Dr. Hastie who is the maintainer of the glmnet package and got the following answer:
In the traditional case, the base category is arbitrary.
In ...
- 689
9
votes
Why is cv.glmnet returning absurd coefficients when intercept term is omitted?
I can explain what you're seeing but not necessarily why it is the way it is. glmnet is starting the no-intercept solution at a much higher initial regularization ...
- 1,673
9
votes
GLMNET: Weights and imbalanced data
Yes, you should provide weights. I assign weights $1 - \frac{\text{# of class members}}{\text{# of total members}}$. Glmnet rescales them to sum to the total number of class members anyway.
Here's an ...
- 352
8
votes
Glmnet: How to select Lambda and Alpha
It appears that the default in glmnet is to select lambda from a range of values from min.lambda to max.lambda, then the optimal ...
- 221
8
votes
Accepted
How can a glmnet model with no coefficients have perfect performance?
It looks like in your data there is no relationship between the covariates and the outcome. I imagine that the model is discovering that and shrinks the coefficients to 0. If you fit on all the data ...
- 39.6k
8
votes
Accepted
Adaptive LASSO, confidence interval and sample size
The penalization of coefficients with methods like lasso, adaptive lasso, and ridge regression means that you can model data even when the number of predictors exceeds the number of observations. You ...
- 102k
8
votes
Accepted
How to implement Adaptive Lasso penalty for a Logistic regression in Python?
Adaptive LASSO is a two-step estimator; check out section 3.1 of Zou "The Adaptive Lasso and Its Oracle Properties" (2006). (This is the original paper that proposed adaptive LASSO.) You can ...
- 69.5k
8
votes
Too good to be true? Ridge prediction
Split sample validation can require up to 20,000 observations to perform well enough. Otherwise the results may depend dramatically on the luck of the split. That’s why 100 repeats of 10-fold cross-...
- 98.5k
7
votes
Accepted
How to report RMSE of Lasso using glmnet in R
Specifically, do I report the RMSE of the model itself (i.e., how it performs with the training data used to create it) or do I report the RMSE of the model's performance with new data (aka test data)?...
- 20.6k
7
votes
Accepted
Does R glmnet regularize on intercept?
It does not penalize the intercept. But it does penalize the covariates, which are correlated with the intercept. Thus, changing the estimates of the coefficients for the non-constant variables ...
- 21.6k
7
votes
Why does glmnet use coordinate descent for Ridge regression?
I think this is due to speed. Cyclical coordinate descent does not find the exact solution in finite time, but it is faster, not only for a grid of $\lambda$'s but also for a single $\lambda$.
...
- 156
Only top scored, non community-wiki answers of a minimum length are eligible