I have a large unbalanced dataset (the target has ~1500x more 0's than 1's) on which I train a logistic regression algorithm to predict the probability of success (Not a binary outcome but a real number between 0 and 1).

Some additional details:

  • about 100,000,000 lines in my dataset
  • I'm using Online Logistic Regression with a Stochastic Gradient Descent.
  • I apply a decay factor to my learning rate

The average value of the target variable on the training set is about 0.0005. the average of the target variable in the test set is almost exactly the same as in the training set (~0.0005).

However, when I test my model on the test set, the overall average of my predictions is a bit out (~0.001). Should I expect from logistic regression to centre the results around the mean? I know that the main purpose of the intercept term is to do so, but what level of difference should I expect? Is the difference in observed vs. predicted mean negligible here?

  • 1
    $\begingroup$ Related: Properties of logistic regressions $\endgroup$
    – cardinal
    Commented Aug 12, 2014 at 23:19
  • $\begingroup$ I almost always refer people to this paper by Steyerberg: ncbi.nlm.nih.gov/pmc/articles/PMC3575184 Does a better job than me in explaining "Calibration in the Large". But for a logistic model the predicted "Calibration in the Large" will always be same as sample in development set and different in test set. There is a statistical test for significance of the difference, but would need to do some searching to find. $\endgroup$
    – charles
    Commented Aug 13, 2014 at 0:28
  • $\begingroup$ The intercept is the same as the mean only when all predictors are zero. $\endgroup$
    – purple51
    Commented Aug 13, 2014 at 5:15
  • $\begingroup$ Thanks for your comment. Do you have any idea what can be the cause of this difference between the mean in predicted Vs. observed. Can a "bad" learning rate" (which would prevent convergence to occur) explain that? $\endgroup$
    – Aymen
    Commented Aug 13, 2014 at 17:20
  • 1
    $\begingroup$ What precisely are you averaging? Logistic regression produces estimated log odds but it does not specifically produce 0-1 predictions. The mean of log odds is not the same as the log odds of the mean. $\endgroup$
    – whuber
    Commented Aug 13, 2014 at 17:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.