I am working on a logistic regression model that attempts to predict failure events in a population of devices using the previous 10 days of data (60 features from sensor data). The failure event is rare; I would expect to find approximately 20-30 on a given day, and the total population size is > 10,000.

Here is where I may be mistaken: I have access to years-worth of historical data, so I thought that I would be slick and make the event 'non-rare'. That is, collect 1000 failure examples and 1000 non-failure examples, then estimate the model on that. This, I thought, would give the model enough information to determine the relationships between sensor readings and class membership.

However, I am starting to think that my intercept term (and others?) may be problematic because the real initial class probabilities are not respected.

Is the model corrupted by this disregard of the real event probabilities?

Thank you!

  • $\begingroup$ If I'm reading this correctly, this sounds akin to a case-control study in epidemiology/biostatistics -- that is, sampling on event status (event/non-event) and then considering differences in exposure status. This is useful for modelling odds ratios and answering causal questions (e.g. is paste exposure to Substance X more common amongst brain cancer patients than controls?) but less useful if you're looking at predicting event status based on other features (and it sounds like your data structure is more complex than this simple example anyway.) $\endgroup$ – James Stanley Sep 15 '15 at 22:30
  • $\begingroup$ See Does down-sampling change logistic regression coefficients?. $\endgroup$ – Scortchi Sep 18 '15 at 11:54

What you do is 'over sample' on the 'failures' and as you conclude, this biases your estimates. When you do not draw a random sample, but a sample based on the values of the binary outcome, then you are doing a so-called 'choice based sampling'.

The correction of King and Zeng is one option, another option is to use the weights parameter in glm: If $\pi$ is the fraction of failures in your population and $p$ the fraction in the sample, then each observation gets a weight which is $\frac{\pi}{p}$ for the failures and $\frac{1-\pi}{1-p}$ for the non-failures. For details see e.g. W.H. Greene, Econometric Analysis. Note that in the latter case you should use White's sandwich estimator for the var-covar matrix as was mentioned by @Scortchi in the comments.

  • $\begingroup$ (-1) The glm manual says "For a binomial GLM prior weights are used to give the number of trials when the response is the proportion of successes". So these are frequency weights, not sampling weights: standard errors for odds ratio estimates will be hugely inflated. (BTW I hope it doesn't seem like I'm picking on you today - you should've also noticed a few up-votes.) $\endgroup$ – Scortchi Sep 18 '15 at 12:46
  • $\begingroup$ @Scortchi: no hard feelings, if I am wrong I am wrong, but in this case I am not so sure whether I am, did you check the formula in the reference ? Because if you look at it, these are the weights of the observations in the likelihood function that is maximised ? $\endgroup$ – user83346 Sep 18 '15 at 13:19
  • $\begingroup$ Which reference? $\endgroup$ – Scortchi Sep 18 '15 at 13:47
  • $\begingroup$ @Scortchi: W.H. Greene, Econometric Analysis $\endgroup$ – user83346 Sep 18 '15 at 13:48
  • 1
    $\begingroup$ @Scortchi: You are right about the var-covar estimator, The reference that I menstion (Greene) also says that you should use the ''White's robust sandwich estimator'', Greene refers to White (1982a), Maximum lilekihood estimation of misspecified models, Econometrica 53, 1982, p 1-16. But I think you can no longer say that my answer is wrong ? $\endgroup$ – user83346 Sep 18 '15 at 14:09

This approach definitely does corrupt the model. The output of logistic regression is meant to convey the probability of an event given some configuration of independent variables. Explicitly removing large amounts of data because of the value of the dependent variable will definitely harm this aspect of the model's output.

It sounds like your goal is to classify as opposed to using these probabilities, so if the skew is leading to two many false negatives, you should produce an ROC curve and determine if there is some threshold which produces an acceptable amount of false positives/negatives.

The problem that you discuss has been studied greatly and is often referred to as one-class classification. When the vast majority of the data is "normal" and the goal is to detect "anomalies" (for example flight malfunction) there are many methods which essentially focus on understanding normality as opposed to understanding anomaly.

  • $\begingroup$ From what I am reading, it appears I may be able to estimate the model using my 'enhanced' dataset created by selecting on Y, then correct the intercept manually using the King and Zeng method (base-rate correction). Can you speak to that at all? $\endgroup$ – HEITZ Sep 15 '15 at 21:31
  • $\begingroup$ While I've heard of the King and Zeng method, I do not know it. I may look into it, but can't make any promises. I do strongly suggest you look into methods of "one class classification" though. $\endgroup$ – jlimahaverford Sep 15 '15 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.