Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I would like to study rare events in a finite population. Since I am unsure about which strategy is best suited, I would appreciate tips and references related to this matter, although I am well-aware it has been largely covered. I just don't really know where to begin.

My problem is a political sciences one and I have a finite population comprising 515,843 records. They are associated to a binary dependent variable with 513,334 "0"s and 2,509 "1"s. I can coin my "1"s as rare events since they account for only 0.49% of the population.

I have a set of around 10 independent variables I would like to build a model with to explain the presence of "1"s. Like many of us, I read King & Zeng's 2001 article about rare events correction. Their approach was to use a case-control design to reduce the number of "0"s, then apply correction to the intercept.

However, this post says that King & Zeng's argument was not necessary if I already collected my data over the whole population, which is my case. Therefore, I have to use the classical logit model. Unfortunately for me, although I obtain good significant coefficients, my model is completely useless in terms of prediction (fails to predict 99.48% of my "1"s).

After reading King & Zeng's article, I wanted to try a case-control design and selected only 10% of the "0"s with all the "1"s. With almost the same coefficients, the model was able to predict almost one third of the "1"s when applied to the full population. Of course, there are a lot of false-positive.

I have thus three questions I would like to ask you:

1) If King & Zeng's approach is prejudiciable when you have full knowledge of the population, why do they use a situation where they know the population in their article to prove their point?

2) If I have good and siginificant coefficients in a logit regression, but very poor predictive power, does that mean that the variation explained by these variable is meaningless?

3) What is the best approach to deal with rare events? I read about King's relogit model, Firth's approach, the exact logit, etc. I must confess I am a lost among all these solutions.

Thank you for your help!

share|improve this question
    
The number sounds familiar...by any chance a dataset about ethnic conflict? If yo, it is a time series - I used a survival model to great success in a ethnic conflict study... –  Christian Sauer Jul 11 at 12:39
    
Close enough. It's a dataset about the location of conflict events in Africa. However, I study the location of these events without accounting for time. –  Damien Jul 11 at 12:56
    
Ah, a lot of my cases came from Africa, since ethnic conflicts are rampant there. Do you da geographic study? Would it be a huge problem to account for time? I found it really useful, especially due to the fact that certain variables are changing with time (political system, cold war etc.) –  Christian Sauer Jul 11 at 13:16
    
I am using UCDP's GED dataset which covers the period 1989-2010. I am interested in the geographical factors that can play a role in the location of conflict events. Time variations have certainly a lot to say, but the questions answered are different. Also, many of my independent variables are either unavailable for different periods (land cover) or did not change at all (topography) –  Damien Jul 11 at 13:22
    
Have fun :) Would you be interested in sharing with my results when you have finished? I like to compare it to my results. If I remember correctly, I found no correlation between mountains and ethnic conflict, a thesis which was quite the rage "back then". –  Christian Sauer Jul 11 at 13:27

5 Answers 5

(1) If you've "full knowledge of a population" why do you need a model to make predictions? I suspect you're implicitly considering them as a sample from a hypothetical super-population—see here & here. So should you throw away observations from your sample? No. King & Zeng don't advocate this:

[...] in fields like international relations, the number of observable 1’s (such as wars) is strictly limited, so in most applications it is best to collect all available 1’s or a large sample of them. The only real decision then is how many 0’s to collect as well. If collecting 0’s is costless, we should collect as many as we can get, since more data are always better.

The situation I think you're talking about is the example "Selecting on $Y$ in Militarized Interstate Dispute Data". K.&Z. use it to, well, prove their point: in this example if a researcher had tried to economize by collecting all the 1's & a proportion of the 0's, his estimates would be similar to one who'd sampled all available 1's & 0's. How else would you illustrate that?

(2) The main issue here is the use of an improper scoring rule to assess your model's predictive performance. Suppose your model were true, so that for any individual you knew the probability of a rare event—say being bitten by a snake in the next month. What more do you learn by stipulating an arbitrary probability cut-off & predicting that those above it will be bitten & those below it won't be? If you make the cut-off 50% you'll likely predict no-one will get bitten. If you make it low enough you can predict everyone will get bitten. So what? Sensible application of a model requires discrimination—who should be given the only vial of anti-venom?— or calibration—for whom is it worth buying boots, given their cost relative to that of a snake-bite?.

share|improve this answer
    
Thank you for you answer. Regarding (1), would it be more appropriate to speak about a sample of the observations we know so far to account for the possibility of future events? Regarding (2), I spent a moment trying to figure out what a scoring rule is. If I understand correctly the Wikipedia article, I should vary the scoring function across different values of probability for which the event is expected to happen, then choose as cutoff value the probability which had the highest score. If I choose the logarithmic scoring rule, how am I supposed to implement the expected value? –  Damien Jul 11 at 13:02
    
(1) Yes, imagining they're sampled from a population from which future events will be drawn. (2) Forget about cut-offs. The area under the receiver operating characteristic curve is useful for assessing pure discrimination; for overall performance use a metric that takes the magnitude of the difference between predictions & outcomes into account - say Brier scores (quadratic) or Nagelkerke's $R^2$ (logarithmic). –  Scortchi Jul 14 at 10:47

On one level, I wonder how much of your model's inaccuracy is simply that your process is hard to predict, and your variables aren't sufficient to do so. Are there other variables that might explain more?

On the other hand, if you can cast your dependent variable as a count/ordinal problem (like casualties from conflict, or duration of conflict), you might try zero-inflated count regression or hurdle models. These might have the same issue of poor definition between 0 and 1, but some conflicts that your variables are correlated with could pull away from zero.

share|improve this answer
1  
(+1) Good suggestions. I'd like to harp on though, that the model's "inaccuracy" is merely a failure to predict many probabilities over 50%. If the "1"s typically have predicted probabilities of 10% to 40%, compared with a little under 0.5% for the "0"s - that would be considered strong predictive performance in many applications. –  Scortchi Aug 7 at 9:28

Your question boils down to how can I coax logit regression to find a better solution. But are you even sure that a better solution exists? With only ten parameters, were you able to find a better solution?

I would try a more complicated model by for example adding product terms at the input, or adding a max-out layer on the target side (so that you essentially have multiple logistic regressors for various adaptively discovered subsets of target 1s).

share|improve this answer
    
Thanks for your answer. I will definitely try combining my variables in different ways. But before, I want to know if the poor performances of my model come from technical issues or from somewhere else –  Damien Jul 11 at 13:04

In addition to downsampling the majority population you can oversample the rare events as well, but be aware that oversampling of the minority class may lead to overfitting, so check things carefully.

This paper can give more information about it: Yap, Bee Wah, et al. "An Application of Oversampling, Undersampling, Bagging and Boosting in Handling Imbalanced Datasets." pdf

Also, I'd like to link this question since it discusses the same issue as well

share|improve this answer

Great question.

To my mind, the issue is whether you're trying to do inference (are you interested in what your coefficients are telling you?) or prediction. If the latter, then you could borrow models from Machine Learning (BART, randomForest, boosted trees, etc.) that will almost certainly do a better job at prediction than logit. If you're doing inference, and you have so many datapoints, then try including sensible interaction terms, polynomial terms, etc. Alternatively, you could do inference from BART, as in this paper:

http://artsandsciences.sc.edu/people/kernh/publications/Green%20and%20Kern%20BART.pdf

I have been doing some work recently on rare events, and had no idea beforehand how much rare cases can affect the analysis. Down-sampling the 0-cases is a must. One strategy to find the ideal down-sample proportion would be

  1. Take all your 1s, let's say you have n1 of them.
  2. Set some value z = multiple of the n1 you will draw; perhaps start at 5 and reduce to 1.
  3. draw z*n1 0 observations
  4. Estimate your model on a sample of your subset data, making sure that you cross-validate on the whole dataset
  5. Save the relevant fit measures you're interested in: coefficients of interest, AUC of a ROC curve, relevant values in a confusion matrix, etc.
  6. Repeat steps 2:5 for successively smaller zs. You will probably find that as you down-sample, the false-negative to false positive ratio (in your test-set) will decrease. That is, you'll start predicting more 1s, hopefully that are genuinely 1s, but also many that are actually 0s. If there is a saddle point in this misclassification, then that would be a good down-sample ratio.

Hope this helps. JS

share|improve this answer
    
(-1) No need at all to down-sample for logistic regression. See here; selecting on the response only changes the expected intercept, so down-sampling just reduces precision of the estimated odds ratios. Logistic regression gives you predicted probabilities, which you may use to classify using cut-offs calculated to take into account the costs of different kinds of mis-classification, or use to rank individuals, or be interested in in their own right. –  Scortchi Aug 6 at 18:14
    
You will notice that I didn't mention using logistic regression, and instead suggested that there are methods (like down-sampled BART) that are probably more appropriate for rare cases. –  Jim Aug 7 at 19:27
    
The question's about logistic regression, & whether to down-sample when doing it, & you seem to be discussing logistic regression when you write about "including sensible interaction terms, polynomial terms"; so it isn't clear that your advice on down-sampling is only intended for use with alternative methods: perhaps you'd consider editing your answer to make it clear. –  Scortchi Aug 7 at 23:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.