# Average of predicted values with logistic regression

I have a large unbalanced dataset (the target has ~1500x more 0's than 1's) on which I train a logistic regression algorithm to predict the probability of success (Not a binary outcome but a real number between 0 and 1).

• about 100,000,000 lines in my dataset
• I'm using Online Logistic Regression with a Stochastic Gradient Descent.
• I apply a decay factor to my learning rate

The average value of the target variable on the training set is about 0.0005. the average of the target variable in the test set is almost exactly the same as in the training set (~0.0005).

However, when I test my model on the test set, the overall average of my predictions is a bit out (~0.001). Should I expect from logistic regression to centre the results around the mean? I know that the main purpose of the intercept term is to do so, but what level of difference should I expect? Is the difference in observed vs. predicted mean negligible here?

• Commented Aug 12, 2014 at 23:19
• I almost always refer people to this paper by Steyerberg: ncbi.nlm.nih.gov/pmc/articles/PMC3575184 Does a better job than me in explaining "Calibration in the Large". But for a logistic model the predicted "Calibration in the Large" will always be same as sample in development set and different in test set. There is a statistical test for significance of the difference, but would need to do some searching to find. Commented Aug 13, 2014 at 0:28
• The intercept is the same as the mean only when all predictors are zero. Commented Aug 13, 2014 at 5:15
• Thanks for your comment. Do you have any idea what can be the cause of this difference between the mean in predicted Vs. observed. Can a "bad" learning rate" (which would prevent convergence to occur) explain that? Commented Aug 13, 2014 at 17:20
• What precisely are you averaging? Logistic regression produces estimated log odds but it does not specifically produce 0-1 predictions. The mean of log odds is not the same as the log odds of the mean.
– whuber
Commented Aug 13, 2014 at 17:48