2
$\begingroup$

I have data in which the number of negative cases in response is approximately 98% of the total sample size (total # records are approximately 1 million, Response is binary). The positive cases are roughly 2%. What are the limitations of applying 'glm' and 'cart' on such data? What option do I have in such cases?

On test data I did get a very good AUC ~0.92. How much faith should I have in this model considering such a disparity in the number of cases in the positive and negative categories?

$\endgroup$
  • 2
    $\begingroup$ what was the sample size of positive cases? $\endgroup$ – Peter Flom Oct 29 '13 at 22:22
  • 1
    $\begingroup$ Are you and learner the same person? $\endgroup$ – user88 Oct 30 '13 at 10:16
  • $\begingroup$ So, please, consider merging your two accounts /cc @mbq. $\endgroup$ – chl Oct 30 '13 at 10:28
  • $\begingroup$ Actually I posted my comment when I was in my office where i use my office email. Right now i used my personal computer at home which by default used my personal mail. But thanks for suggestion. I would consider merging these accounts $\endgroup$ – user1140126 Oct 30 '13 at 10:31
2
$\begingroup$

Is this a binary response variable?

One limitation of models with very small (or large) rates is the amount of data needed to get accurate and stable estimates of variance and sample errors. As a rule of thumb you would want both Np = 5 and N(1-p) = 5 (or higher), so with an estimated p of 0.02 you need N of 250. So in order to get accurate sample errors etc you want a minimum 250 observations.

$\endgroup$
  • $\begingroup$ thanks, yes it does. Does it matter if the response is continuous $\endgroup$ – user1140126 Oct 30 '13 at 10:21
  • $\begingroup$ This is not correct, David. Precision of the estimator does not depend on variability in the dependant variable, only the independent variable. $\endgroup$ – Christian Dec 3 '13 at 1:11
2
$\begingroup$

For logistic regression there's no problem with imbalanced samples per se, though if the absolute number in either response class is small for separate covariate patterns the maximum-likelihood estimators of odds ratios can be rather too biased for comfort, and some prefer to used penalized-likelihood methods as discussed here. For larger models computational constraints might necessitate down-sampling the most common class, which reduces the precision of all estimates somewhat but otherwise only affects the intercept (an estimate of which for the original population can be recovered —see here). In your case a minority class numbering 20k shouldn't give cause for concern unless you're trying to estimate odds ratios for some very rare predictor categories.

For classification trees there's what seems to be a good answer here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.