1
$\begingroup$

I have an existing logistic regression used to forecast whether a given action will take place or not. The unit of analysis is the individual person. For the sake of the question, assume a default threshold of p > .5 yields a positive prediction. When I plot the proportion of individuals who are actually in each class for a given day vs. the proportion of individuals predicted to be in each class for a given day, the model drastically underpredicts. For example, the model will predict that 8% of individuals will be in class 1 whereas the actual proportion is 13%.

However, it was suggested that I take the average of the predicted probabilities for all individuals on a given day to predict the proportion of people in each class. This actually works very well (within a % or so).

How is it the case that a model that is individually quite poor is collectively quite good? Averaging predicted probabilities does not seem like a sound solution to me, but I cannot figure out quite why.

$\endgroup$
1
$\begingroup$

This is a comment, but my reputation isn't high enough to comment yet lol. Anywho, how did you build this existing logistic regression model? how large was the sample? I am assuming that Class 1 is that the given action will take place whereas Class 2 would be that the action would not take place? My first impression is that the model is modeling better for the average of the predicted probabilities for all individuals on a given day to predict the proportion of people in each class better than it does for individuals by mere chance (ie. if you were to use this on another 20 samples, this would likely not be the case, as the model was created to predict class membership for individuals). I just wanted to clear up some questions that others and myself have in regards to your question to speed up the process of you receiving the answer you are looking for!

$\endgroup$
1
  • $\begingroup$ The model was built on 80,000 individuals and is being used to predict an out-of-sample 20,000 individuals. I don't think sample size is a concern. I suspect it is because it is a very random process to model and the aggregated probabilities are just reflecting the (fairly stable) base rate. $\endgroup$
    – Trey
    Aug 14 '13 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.