Timeline for Logistic regression using penalized likelihood (lasso?) in Matlab/R
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 26, 2014 at 20:48 | comment | added | Gavin Simpson | @Maddy & I also misspoke (miswrote) about lasso reducing the bias. It tries to reduce variance due to estimating $\beta$s for all covariates at expense of a bit of _increased_ bias in the remaining $\beta_{\mathrm{lasso}}$. If the reduction in variance exceeds the increase in bias, then MSE will be reduced. That's what the Lasso aims to get at. | |
| Mar 26, 2014 at 20:46 | comment | added | Gavin Simpson | @Maddy Oops, brain fail on my part. I was thinking of coordinate descent not gradient descent, and ignore my loss function ramblings. Clearly I had gradient boosting on my mind when I wrote the comment. The idea of using L1, L2 and other penalties in GLMs is well established with fast cyclic coordinate descent algorithms for many GLM models now developed. With The lasso penalty, the idea is that some of the covariates have zero or near zero $\beta$s, i.e. the solution is sparse. If the truth really is sparse, then estimating $\beta$s for all covariates is overfitting | |
| Mar 26, 2014 at 20:29 | comment | added | Maddy | Your comment about lasso & gradient descent has confused me. IIRC, gradient descent is used to calculate the ML coeff. How does lasso get in this? I have used gradient descent algorithm in a machine learning class. Rest of your comment helps me. Thanks. | |
| Mar 26, 2014 at 19:02 | history | edited | Gavin Simpson | CC BY-SA 3.0 |
added 2 characters in body
|
| S Mar 26, 2014 at 18:59 | history | suggested | Andre Silva | CC BY-SA 3.0 |
fixed hyperlink format
|
| Mar 26, 2014 at 18:57 | review | Suggested edits | |||
| S Mar 26, 2014 at 18:59 | |||||
| Mar 26, 2014 at 17:20 | history | edited | Scortchi♦ | CC BY-SA 3.0 |
fixed typo
|
| Mar 26, 2014 at 4:44 | comment | added | Gavin Simpson | @Maddy the lasso can be used in a logistic regression (IIRC you can rewrite the path algorithm as a gradient descent algorithm minimising a given loss function, which for logistic regression includes two options). It is there to reduce bias in the model estimates arising from fitting a large number of parameters. See the glmnet package for R as one implementation. I don't know how it would help in the small number of 1s case though? It doesn't deal with correlated predictors though, unlike ridge regression. The elastic net combines lasso & ridge penalties to handle sparsity & collinearity. | |
| Mar 26, 2014 at 4:14 | comment | added | Maddy |
@gaving simpson You are right. I did read about that elsewhere as well. I up voted this to highlight your comments on Firth's method. Thanks. I'm still wondering if lasso is used to reduce the number of correlated predictor variables or for something else in Logistic Regression. Please let me know.
|
|
| Mar 24, 2014 at 16:42 | history | answered | Gavin Simpson | CC BY-SA 3.0 |