7
$\begingroup$

I was trying to find out whether an oversampling can really make a model better. On this blog page, it says that it can improve a decision tree, but it shouldn't improve a logistic regression. Quotation below:

Standard statistical techniques are insensitive to the original density of the data. So, a logistic regression run on oversampled data should produce essentially the same model as on the original data.

Question:
Is it really true that using an oversampling in a logistic regression will not improve results?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ You should specify if you want to oversample based on covariates or on the outcome. But the general idea is that if your event of interest is very rare, then your results might be biased under random sampling due to finite nature of the sample. A good reference on the bias of a logistic model under rare events can be found here: Gary King's website. $\endgroup$
    – mmgm
    Commented Jan 25, 2012 at 16:35
  • 1
    $\begingroup$ A final note is that i think that what the authors of that post might be telling you is that under biased sampling strategies (outcome based sampling), the Logistic regression will still give you the right odds ratios as long as the event of interest is rare. That is a property of the logistic which proof is provided in Manski's book and that is very used in biomedical sciences due to the cost of gathering data on things that are infrequent. $\endgroup$
    – mmgm
    Commented Jan 25, 2012 at 16:52
  • $\begingroup$ @mmgm: I thought about oversampling on the outcome to balance rare categories. If I understood correctly, when a category is rare then results will be biased, but the order should be all right. You can write this as an asnwer and I will be glad to accept it. $\endgroup$
    – Tomek Tarczynski
    Commented Jan 26, 2012 at 7:48

1 Answer 1

Reset to default
9
$\begingroup$

If you are thinking about oversampling based on the outcome then you have to be quite careful.

In general this is not ok. Endogenous sampling schemes will give you biased results. See Wooldridge chapter 17.

In the case of the Logit model things are somewhat different. Borrowing from Manski the thing is that in general response based sampling reveals: $P(x|y=1)$ and $P(x|y=0)$, but nothing about the probability of $P(y|x)$.

However, if $P(x|y=1)>0$ and $P(x|y=0)>0$ and also $P(y=1|x)$ approaches zero (in other words, it has the property of the rare disease assumption), then for the LOGIT model (due to its nice exponential functional form), outcome based sampling point identifies relative and attributable risk. Relative risk is also known as the odds ratio when the rare disease assumption is true. For the complete proof check pages 112 and 113 of Manski's book that I recommended above.

OK, so we have established that if you are only looking at the odds ratios, outcome based sampling and logit should be OK. Of course, you can only get odds ratios, you won't be able to identify the intercept therefore you must be very very careful about what to conclude ...

Now to your question:

In principle, if you have the entire population, you should not need to oversample since you have all cases ... That should be clear.

In practice that is rarely true since most likely you will only have a sample of the population. When that happens, what Gary King shows is that there are some benefits on doing outcome based sampling and applying finite sample corrections: Gary King Rare Events Logit.

Finally (without having read it in detail) I believe that the point of the blog that you are referring to above is precisely the one that I am making here. That if you have no other alternative than to oversample a class of the population based on the outcome to get a minimum amount of observations, then, with the logistic regression and if you are looking at the odds ratio you should be fine. It is not that they recommend it, but it might be your only option.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Are there any numerical heuristics about when you would want to apply something like King's procedure? Does the Manski book have something like this. For example, should I worry about the logit if my population has 40 million observations, with a 1% rate of $Y=1$ and I take a 10% sample? $\endgroup$
    – dimitriy
    Commented Nov 14, 2012 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.