# Logistic regression - Empty test sample doesn't return probability of 0.5

I've just started with machine learning. I managed to build logistic regression model with 10 features. All features are feature scaled and centered around 0. Train/test ratio is around 0.3.

My code in Python:

# Training set
imputer_train = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_train = imputer_train.fit(self.X_train)
self.X_train = imputer_train.transform(self.X_train)

# Test set
imputer_test = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_test = imputer_test.fit(self.X_test)
self.X_test = imputer_test.transform(self.X_test)

# Feature scaling
self.sc_X = StandardScaler()
self.X_train = self.sc_X.fit_transform(self.X_train)
self.X_test = self.sc_X.transform(self.X_test)

# Fitting Logistic Regression to the Training set
from sklearn.linear_model import LogisticRegression
self.classifier = LogisticRegression()
self.classifier.fit(self.X_train, self.y_train)

self.y_pred = self.classifier.predict(self.X_test)
self.y_pred_proba = self.classifier.predict_proba(self.X_test)[:, 1]

logistic_loss = log_loss(y_true=self.y_test, y_pred=self.y_pred_proba)
print "Logistic loss: %2.3f%%" % (logistic_loss * 100)


Model summary:

Model correctness: ~94%
Logistic loss: ~14%


The problem:

When I try to test my model with single test which consist of

X_test = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]


I get my y_pred around 0.43.

Another example:

If I calculate features for a single event like this:

X_test = [0.113591, 0.044682, 0.41523, -0.00565, 0.063624, 1.159652, 0, 0.090699, 0.184688, 0]


I get y_pred = 0.57806259.

And then I invert calculation of the same event I get features:

X_test_inverted = [-0.113591, -0.044682, -0.41523, 0.00565, -0.063624, -1.159652, 0, -0.090699, -0.184688, 0]


I get y_pred = 0.27111127.

Is this normal that probability of inverted events dont sum up to 1? Why is this so?

If not, where can I start to fix that problem?

Thank you.

It is normal. The output probability of the model is, in fact, $p(y=1 | x; \theta)$, where $\theta$ is the set of fitted parameters of the model, and $x$ is the vector of features. There is no reason why

$p(y=1 | x; \theta) + p(y=1 | -x; \theta)$

should sum to 1.

• Thanks for your solution. With your help I'm understanding what's going on. :) – matox May 16 '17 at 11:15
• @matox No problem. If you want to understand the probabilistic interpretation of the algorithm, I suggest cs229.stanford.edu/notes/cs229-notes1.pdf. Not too heavy, so you don't get lost if you are a beginner. – alebu May 16 '17 at 15:47

If you consider logistic regression to be learning parameters $\beta_i$ for the function $$f(\beta_0+\sum_{i=1}^{10}\beta_ix_i)$$ were $f$ transforms the linear sum to a probability range between 0 and 1. I think you may be failing to consider the $\beta_0$ term.

When x_test = [0,0..] therefore $x_i=0$ $$x_i = 0$$ $$y = f(\beta_0) = 0.43$$ In your case with the $\beta_0$ learning by the algorithm.

• I really like your explanation. It make sense now. Thank you. :) – matox May 16 '17 at 11:13
• No problem, glad it was useful – Harry Salmon May 16 '17 at 13:25

Your test sample that is all zeros (x=[0,0,0,..]) is not an empty test sample, but a sample with all features set at the mean, given that you state that all features were centered. Thus, the p = .43 is perfectly reasonable and reflects the predicted probability when all features are zero.

• Thank you for you input, really appreciate it. I'm going to upvote it when I have enough of reputation. Thanks again :) – matox May 16 '17 at 11:11