I am trying to use logistic regression in a scenario where there are very few positives. I'm aware that maximum likelihood suffers from small sample bias. So MATLAB's glmfit doesn't work for me. I tried using firth regression in R but it simply hangs up my powerful PC (150,000 observations, 9 dummy variables, 1500 positives). MATLAB also has the function lassoglm but I'm not sure if it can be used for logistic regression with few positives. Can you please confirm/suggest alternative?
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ Your concern about ML small sample bias is not justified given your number of observations. $\endgroup$– JohnCommented Mar 24, 2014 at 16:24
-
$\begingroup$ @John I did use LR using gmlfit in Matlab. It gave me statistically significant coeff but the computed probability for an event was ~1%. In my experience, a good LR model can give you calculated probabilities of say 60%, 80%, 20% and you generally pick a threshold as 50% to mark these as 1s and 0s. This didn't work for me as my calculated probabilities were max of 1%. Should I lower the threshold or should I use penalized LR? Thanks. $\endgroup$– MaddyCommented Mar 24, 2014 at 16:33
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
Given the stated sample size (150,000), why do you think there will be bias due to "small sample" size? You have 1500 observations of the positive case, which I wouldn't normally think of as being small.
An alternative in R is the brglm package, which also implements Firth's method.
However, I was recently pointed to a paper (Gelman et al 2008) which showed Firth's method performing quite badly compared to other methods for bias reduction, in a Bayesian context. The authors of that paper wrote the bayesglm() function in R package arm to provide a range of priors on the model coefficients (Firth's method boils down to a particular choice of prior in the model; a Jeffreys prior). My main reason for mentioning it is that it may also work more efficiently than the options considered thus far.
answered Mar 24, 2014 at 16:42
-
$\begingroup$ @gaving simpson You are right. I did read about that elsewhere as well. I up voted this to highlight your comments on Firth's method. Thanks. I'm still wondering if
lassois used to reduce the number of correlated predictor variables or for something else in Logistic Regression. Please let me know. $\endgroup$– MaddyCommented Mar 26, 2014 at 4:14 -
$\begingroup$ @Maddy the lasso can be used in a logistic regression (IIRC you can rewrite the path algorithm as a gradient descent algorithm minimising a given loss function, which for logistic regression includes two options). It is there to reduce bias in the model estimates arising from fitting a large number of parameters. See the glmnet package for R as one implementation. I don't know how it would help in the small number of 1s case though? It doesn't deal with correlated predictors though, unlike ridge regression. The elastic net combines lasso & ridge penalties to handle sparsity & collinearity. $\endgroup$ Commented Mar 26, 2014 at 4:44
-
$\begingroup$ Your comment about lasso & gradient descent has confused me. IIRC, gradient descent is used to calculate the ML coeff. How does lasso get in this? I have used gradient descent algorithm in a machine learning class. Rest of your comment helps me. Thanks. $\endgroup$– MaddyCommented Mar 26, 2014 at 20:29
-
$\begingroup$ @Maddy Oops, brain fail on my part. I was thinking of coordinate descent not gradient descent, and ignore my loss function ramblings. Clearly I had gradient boosting on my mind when I wrote the comment. The idea of using L1, L2 and other penalties in GLMs is well established with fast cyclic coordinate descent algorithms for many GLM models now developed. With The lasso penalty, the idea is that some of the covariates have zero or near zero $\beta$s, i.e. the solution is sparse. If the truth really is sparse, then estimating $\beta$s for all covariates is overfitting $\endgroup$ Commented Mar 26, 2014 at 20:46
-
$\begingroup$ @Maddy & I also misspoke (miswrote) about lasso reducing the bias. It tries to reduce variance due to estimating $\beta$s for all covariates at expense of a bit of _increased_ bias in the remaining $\beta_{\mathrm{lasso}}$. If the reduction in variance exceeds the increase in bias, then MSE will be reduced. That's what the Lasso aims to get at. $\endgroup$ Commented Mar 26, 2014 at 20:48
$\begingroup$
$\endgroup$
With 1% positives, you have unbalanced classes. You do not have a small-sample problem as some have pointed out. Instead, you have a rare-event scenario (aka class imbalance). Logistic regression optimized with vanilla MLE will be dominated by the major class (99%).
In such a case, alternatives to vanilla logistic regression include:
- Rare events logistic regression (
Zelig::relogitin R implementing King, Leng 2001) which uses weighting and bias correction to address the imbalance. - Firth regression which uses a penalized MLE instead. (
brglmand the newerbrglm2may be faster implementations.)
Note that the lasso penalty reduces the model dimensionality and may help with MLE convergence. However, it does not address the biases driven by the class imbalance.
