I have data in which the number of negative cases in response is approximately 98% of the total sample size (total # records are approximately 1 million, Response is binary). The positive cases are roughly 2%. What are the limitations of applying 'glm' and 'cart' on such data? What option do I have in such cases?
On test data I did get a very good AUC ~0.92. How much faith should I have in this model considering such a disparity in the number of cases in the positive and negative categories?
Is this a binary response variable?
One limitation of models with very small (or large) rates is the amount of data needed to get accurate and stable estimates of variance and sample errors. As a rule of thumb you would want both Np = 5 and N(1-p) = 5 (or higher), so with an estimated p of 0.02 you need N of 250. So in order to get accurate sample errors etc you want a minimum 250 observations.
For logistic regression there's no problem with imbalanced samples per se, though if the absolute number in either response class is small for separate covariate patterns the maximum-likelihood estimators of odds ratios can be rather too biased for comfort, and some prefer to used penalized-likelihood methods as discussed here. For larger models computational constraints might necessitate down-sampling the most common class, which reduces the precision of all estimates somewhat but otherwise only affects the intercept (an estimate of which for the original population can be recovered —see here). In your case a minority class numbering 20k shouldn't give cause for concern unless you're trying to estimate odds ratios for some very rare predictor categories.
For classification trees there's what seems to be a good answer here.
