Logistic regression - Empty test sample doesn't return probability of 0.5

Question

I've just started with machine learning. I managed to build logistic regression model with 10 features. All features are feature scaled and centered around 0. Train/test ratio is around 0.3.

My code in Python:

# Training set
imputer_train = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_train = imputer_train.fit(self.X_train)
self.X_train = imputer_train.transform(self.X_train)

# Test set
imputer_test = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_test = imputer_test.fit(self.X_test)
self.X_test = imputer_test.transform(self.X_test)

# Feature scaling
self.sc_X = StandardScaler()
self.X_train = self.sc_X.fit_transform(self.X_train)
self.X_test = self.sc_X.transform(self.X_test)

# Fitting Logistic Regression to the Training set
from sklearn.linear_model import LogisticRegression
self.classifier = LogisticRegression()
self.classifier.fit(self.X_train, self.y_train)

self.y_pred = self.classifier.predict(self.X_test)
self.y_pred_proba = self.classifier.predict_proba(self.X_test)[:, 1]

logistic_loss = log_loss(y_true=self.y_test, y_pred=self.y_pred_proba)
print "Logistic loss: %2.3f%%" % (logistic_loss * 100)

Model summary:

Model correctness: ~94%
Logistic loss: ~14%

The problem:

When I try to test my model with single test which consist of

X_test = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I get my y_pred around 0.43.

Another example:

If I calculate features for a single event like this:

X_test = [0.113591, 0.044682, 0.41523, -0.00565, 0.063624, 1.159652, 0, 0.090699, 0.184688, 0]

I get y_pred = 0.57806259.

And then I invert calculation of the same event I get features:

X_test_inverted = [-0.113591, -0.044682, -0.41523, 0.00565, -0.063624, -1.159652, 0, -0.090699, -0.184688, 0]

I get y_pred = 0.27111127.

Is this normal that probability of inverted events dont sum up to 1? Why is this so?

If not, where can I start to fix that problem?

Thank you.

Answer 1

It is normal. The output probability of the model is, in fact, $p(y=1 | x; \theta)$, where $\theta$ is the set of fitted parameters of the model, and $x$ is the vector of features. There is no reason why

$p(y=1 | x; \theta) + p(y=1 | -x; \theta)$

should sum to 1.

Answer 2

If you consider logistic regression to be learning parameters $\beta_i$ for the function $$ f(\beta_0+\sum_{i=1}^{10}\beta_ix_i) $$ were $f$ transforms the linear sum to a probability range between 0 and 1. I think you may be failing to consider the $\beta_0$ term.

When x_test = [0,0..] therefore $x_i=0$ $$x_i = 0$$ $$y = f(\beta_0) = 0.43$$ In your case with the $\beta_0$ learning by the algorithm.

Answer 3

Your test sample that is all zeros (x=[0,0,0,..]) is not an empty test sample, but a sample with all features set at the mean, given that you state that all features were centered. Thus, the p = .43 is perfectly reasonable and reflects the predicted probability when all features are zero.