3
$\begingroup$

I have a dataset of 600 observations and the target variable is binary(risky/ non-risky). The constraint I have while modelling is that the target variable is very imbalanced .i.e only 12 (only 2% incidence rate) of these observations are risky and others are non risky . I have gone through literature on rare events and have found out that logistic regression for rare events might produce biased estimates if the number of observations with incidence is low (in my case its 12)

To overcome this bias in the estimating process, people started to use penalized regrssion like firth regression (if incidence rate is low but number of observations with incidence is higher, like 2% of 100000 observations is 2000) . And in the case of the sample size and incidence rate both being low(like 3-6%), literature suggests to use something called as "Exact Logistic Regression" which use MCMC sampling.

Since I do my modelling generally in R, I had found tutorials wherein they had used "elrm" package in R to fit logistic regression. But unfortunately "elrm" package has been removed from the R Cran - Repository. Hence that made me question the credibility of the Exact Logistic Regression for "Small Sample Size Rare Events(incidence rate around 3-8%)". It would be of great help to me , if you folks could provide me some insights on the working mechanism of Exact Logistic Regression and whether should I be using regular Logistic Regression (non-penalized estimation) or Exact Logistic Regression (penalized estimation) for my dataset of 600 observations and 2% incidence rate.

It would also be of great help to me if you could provide me insights on the methods ( apart from firth regression and Exact Logistic Regression)which are generally used to model rare events for small sample size?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Could you please give us some more context? What kind of events? How many covariables? with 12 cases you could fit about one predictor, more will be difficult. $\endgroup$ – kjetil b halvorsen Jul 2 '18 at 13:39
  • $\begingroup$ Hey Halvorsen , The data is in the context of loan default.I have around 100 covariables. By event i mean an observation. so 12 events in here means that out of 600 observations I have only 12 of them are risky and and 588 are non risky. Since I have a lot of covariables, does it make sense to use LASSO to predict the outcome variable. Or should i use the exact logistc regression like Bjorn had suggested below? $\endgroup$ – sarath chandra c Jul 3 '18 at 11:02
  • 1
    $\begingroup$ About 100 covariables is way to much! You should start thinking about which might be really important, and which not. There must be some having experience in the field. Otherwise, data reduction is necessary, and that would be lasso or ridge. I am not sure which would be best here! But before anything, try to get more data. And investigate if the covariables are colinear, do a principal component analysis. $\endgroup$ – kjetil b halvorsen Jul 3 '18 at 12:01
  • $\begingroup$ Related: stats.stackexchange.com/a/134385/61496 $\endgroup$ – Yair Daon Jul 6 '18 at 5:40
1
$\begingroup$

One obvious possibility for dealing with having very little information in the dataset you analyze is to use Bayesian methods, particularly if you have prior information on the question under analysis. In most programming languages there are packages that allow you to do that (e.g. rstan, or perhaps easier rstanarm or brms in R, proc mcmc in SAS). In fact, the Firth penalized likelihood regression is equivalent to Bayesian maximum a-posteriori estimation with Jeffreys prior.

Note that without informative priors you will struggle to do much with very sparse data (such as just 12 cases out of 600), unless you are only investigating a single factor that is associated with a huge effect size.

By the way, exact logistic regression does not normally use MCMC sampling, you simply can approximate it quite well using MCMC sampling. Perhaps the impression that this is the standard approach arises, because there simply is no R package that implements any other approach? In contrast e.g. SAS or StatXact have "real" exact logistic regression.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks a lot Bjorn, for taking out your time and explaing this. I have one more question. Lets suppose I want to build a model now using logistic regression for rare data whose incidence rate I know (lets say 2% same as above). Now, an obvious question which flags on my head is the minimum sample size required to get reliable estimates. To get a minimum sample size for rare event data, if I plug in Cochron equation then the value we get is very low and is definitely not representative of the sample size. In such cases, may I know how minium sample size is estimated for rare even data ? $\endgroup$ – sarath chandra c Jul 2 '18 at 17:10
  • $\begingroup$ Your uncertainty will be the greater, the less data you have. How much data you want will depend on your desired amount of information (e.g. Width of a confidence interval or credible interval). $\endgroup$ – Björn Jul 2 '18 at 17:15
  • $\begingroup$ Lets say I would like the Confidence interval to be 1% and confidence level to be 95%). Is there any mathematical equation like cochran equation which gives us a bound or estimate on the minimum sample size for a desired amount of information? $\endgroup$ – sarath chandra c Jul 2 '18 at 17:32
  • $\begingroup$ Confidence intervals would typically be asymmetrical on the probability scale and change width depending on the true value, but approximately symmetrical on the log-odds scale. On that scale you could look at the standard error given the expected cell probabilities, as long as you only have 2 categories or a 2 by 2 table. For those cases these formulae are well known (Google standard error log-odds or log-odds ratio). $\endgroup$ – Björn Jul 2 '18 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.