To reduce the load on the machine I want to take the benefit of the undersampling approach. Here are a few facts about my data:

  1. My data is of the order of 20 millions or even more.

  2. The event rate is around 0.8%.

Using the undersampling approach, I want to reduce the non-event rate and make it either 70:30 or 50:50. This will help me reduce the load on my machine.

I require an accurate estimate of the dependent variable and it is not required only in terms of its order. Hence, I definitely want to calibrate the probabilities back.

Now, two questions:

  1. Using Firth logistic regression (along with the weight statement) will help in this case or not?

  2. Does undersampling help to build a better model in terms of more accurate regression coefficients? I think that all it should change is the intercept and the rest should be the same.

  • 1
    $\begingroup$ I think this is answered here. (1) If you want to reduce bias you can use Firth's method, but there's not necessarily a lot of it just because you've under-sampled. (2) The regression coefficients will be less precisely estimated in a smaller sample, but still consistent estimators of the populations odds ratios; except for, as you say, the intercept. $\endgroup$ Nov 22, 2013 at 9:14
  • $\begingroup$ So decide how much data your machine can comfortably deal with, take all the event data, & make up the difference with non-event data. Fit the logistic regression model (using Firth's penalization if you want), recalculate the intercept according to the population prevalence, & you're done. $\endgroup$ Nov 22, 2013 at 9:36
  • $\begingroup$ Thanks for the response! Actually I have 12 months worth of data. and I was thinking of taking the complete data for the recent two months and undersample the data for the rest of the previous 10 months. 1. Do you think that Firth's method will help me here? or rather where exactly should i use firth's method? $\endgroup$ Nov 22, 2013 at 9:56
  • $\begingroup$ That's a little odd: if you think that recency makes a difference consider modelling it explicitly. As for when to use Firth's method, how much bias there is in the log odds ratio estimates depends on the predictor patterns - you may want to use it if you've some rare classes. $\endgroup$ Nov 22, 2013 at 10:25
  • $\begingroup$ Okay I got it! then another question is that undersampling is helpful only to make the computation easier for the CPU resources. If I have to choose between two options: 1. taking full universe but only for the recent 4 months versus taking the undersampled universe but for two years. Which one should be preferred, I know there is no rule of thumb. But what should be the guidelines to select the better of the two? $\endgroup$ Nov 26, 2013 at 5:23


Browse other questions tagged or ask your own question.