Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I would like to study rare events in a finite population. Since I am unsure about which strategy is best suited, I would appreciate tips and references related to this matter, although I am well-aware it has been largely covered. I just don't really know where to begin.

My problem is a political sciences one and I have a finite population comprising 515,843 records. They are associated to a binary dependent variable with 513,334 "0"s and 2,509 "1"s. I can coin my "1"s as rare events since they account for only 0.49% of the population.

I have a set of around 10 independent variables I would like to build a model with to explain the presence of "1"s. Like many of us, I read King & Zeng's 2001 article about rare events correction. Their approach was to use a case-control design to reduce the number of "0"s, then apply correction to the intercept.

However, this post says that King & Zeng's argument was not necessary if I already collected my data over the whole population, which is my case. Therefore, I have to use the classical logit model. Unfortunately for me, although I obtain good significant coefficients, my model is completely useless in terms of prediction (fails to predict 99.48% of my "1"s).

After reading King & Zeng's article, I wanted to try a case-control design and selected only 10% of the "0"s with all the "1"s. With almost the same coefficients, the model was able to predict almost one third of the "1"s when applied to the full population. Of course, there are a lot of false-positive.

I have thus three questions I would like to ask you:

1) If King & Zeng's approach is prejudiciable when you have full knowledge of the population, why do they use a situation where they know the population in their article to prove their point?

2) If I have good and siginificant coefficients in a logit regression, but very poor predictive power, does that mean that the variation explained by these variable is meaningless?

3) What is the best approach to deal with rare events? I read about King's relogit model, Firth's approach, the exact logit, etc. I must confess I am a lost among all these solutions.

Thank you for your help!

share|improve this question
    
The number sounds familiar...by any chance a dataset about ethnic conflict? If yo, it is a time series - I used a survival model to great success in a ethnic conflict study... –  Christian Sauer Jul 11 at 12:39
    
Close enough. It's a dataset about the location of conflict events in Africa. However, I study the location of these events without accounting for time. –  Damien Jul 11 at 12:56
    
Ah, a lot of my cases came from Africa, since ethnic conflicts are rampant there. Do you da geographic study? Would it be a huge problem to account for time? I found it really useful, especially due to the fact that certain variables are changing with time (political system, cold war etc.) –  Christian Sauer Jul 11 at 13:16
    
I am using UCDP's GED dataset which covers the period 1989-2010. I am interested in the geographical factors that can play a role in the location of conflict events. Time variations have certainly a lot to say, but the questions answered are different. Also, many of my independent variables are either unavailable for different periods (land cover) or did not change at all (topography) –  Damien Jul 11 at 13:22
    
Have fun :) Would you be interested in sharing with my results when you have finished? I like to compare it to my results. If I remember correctly, I found no correlation between mountains and ethnic conflict, a thesis which was quite the rage "back then". –  Christian Sauer Jul 11 at 13:27

2 Answers 2

(1) If you've "full knowledge of a population" why do you need a model to make predictions? I suspect you're implicitly considering them as a sample from a hypothetical super-population—see here & here. So should you throw away observations from your sample? No. King & Zeng don't advocate this:

[...] in fields like international relations, the number of observable 1’s (such as wars) is strictly limited, so in most applications it is best to collect all available 1’s or a large sample of them. The only real decision then is how many 0’s to collect as well. If collecting 0’s is costless, we should collect as many as we can get, since more data are always better.

The situation I think you're talking about is the example "Selecting on $Y$ in Militarized Interstate Dispute Data". K.&Z. use it to, well, prove their point: in this example if a researcher had tried to economize by collecting all the 1's & a proportion of the 0's, his estimates would be similar to one who'd sampled all available 1's & 0's. How else would you illustrate that?

(2) The main issue here is the use of an improper scoring rule to assess your model's predictive performance. Suppose your model were true, so that for any individual you knew the probability of a rare event—say being bitten by a snake in the next month. What more do you learn by stipulating an arbitrary probability cut-off & predicting that those above it will be bitten & those below it won't be? If you make the cut-off 50% you'll likely predict no-one will get bitten. If you make it low enough you'll likely predict everyone will get bitten. So what? Sensible application of a model requires discrimination—who should be given the only vial of antivenom?— or calibration—for whom is it worth buying boots, given their cost relative to that of a snake-bite?.

share|improve this answer
    
Thank you for you answer. Regarding (1), would it be more appropriate to speak about a sample of the observations we know so far to account for the possibility of future events? Regarding (2), I spent a moment trying to figure out what a scoring rule is. If I understand correctly the Wikipedia article, I should vary the scoring function across different values of probability for which the event is expected to happen, then choose as cutoff value the probability which had the highest score. If I choose the logarithmic scoring rule, how am I supposed to implement the expected value? –  Damien Jul 11 at 13:02
    
(1) Yes, imagining they're sampled from a population from which future events will be drawn. (2) Forget about cut-offs. The area under the receiver operating characteristic curve is useful for assessing pure discrimination; for overall performance use a metric that takes the magnitude of the difference between predictions & outcomes into account - say Brier scores (quadratic) or Nagelkerke's $R^2$ (logarithmic). –  Scortchi Jul 14 at 10:47

Your question boils down to how can I coax logit regression to find a better solution. But are you even sure that a better solution exists? With only ten parameters, were you able to find a better solution?

I would try a more complicated model by for example adding product terms at the input, or adding a max-out layer on the target side (so that you essentially have multiple logistic regressors for various adaptively discovered subsets of target 1s).

share|improve this answer
    
Thanks for your answer. I will definitely try combining my variables in different ways. But before, I want to know if the poor performances of my model come from technical issues or from somewhere else –  Damien Jul 11 at 13:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.