I've just started with machine learning. I managed to build logistic regression model with 10 features. All features are feature scaled and centered around 0. Train/test ratio is around 0.3.

My code in Python:

# Training set
imputer_train = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_train = imputer_train.fit(self.X_train)
self.X_train = imputer_train.transform(self.X_train)

# Test set
imputer_test = Imputer(missing_values='NaN', strategy='mean', axis=0)
imputer_test = imputer_test.fit(self.X_test)
self.X_test = imputer_test.transform(self.X_test)

# Feature scaling
self.sc_X = StandardScaler()
self.X_train = self.sc_X.fit_transform(self.X_train)
self.X_test = self.sc_X.transform(self.X_test)

# Fitting Logistic Regression to the Training set
from sklearn.linear_model import LogisticRegression
self.classifier = LogisticRegression()
self.classifier.fit(self.X_train, self.y_train)

self.y_pred = self.classifier.predict(self.X_test)
self.y_pred_proba = self.classifier.predict_proba(self.X_test)[:, 1]

logistic_loss = log_loss(y_true=self.y_test, y_pred=self.y_pred_proba)
print "Logistic loss: %2.3f%%" % (logistic_loss * 100)

Model summary:

Model correctness: ~94%
Logistic loss: ~14%

The problem:

When I try to test my model with single test which consist of

X_test = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I get my y_pred around 0.43.

Another example:

If I calculate features for a single event like this:

X_test = [0.113591, 0.044682, 0.41523, -0.00565, 0.063624, 1.159652, 0, 0.090699, 0.184688, 0]

I get y_pred = 0.57806259.

And then I invert calculation of the same event I get features:

X_test_inverted = [-0.113591, -0.044682, -0.41523, 0.00565, -0.063624, -1.159652, 0, -0.090699, -0.184688, 0]

I get y_pred = 0.27111127.

Is this normal that probability of inverted events dont sum up to 1? Why is this so?

If not, where can I start to fix that problem?

Thank you.


It is normal. The output probability of the model is, in fact, $p(y=1 | x; \theta)$, where $\theta$ is the set of fitted parameters of the model, and $x$ is the vector of features. There is no reason why

$p(y=1 | x; \theta) + p(y=1 | -x; \theta)$

should sum to 1.

  • $\begingroup$ Thanks for your solution. With your help I'm understanding what's going on. :) $\endgroup$ – matox May 16 '17 at 11:15
  • $\begingroup$ @matox No problem. If you want to understand the probabilistic interpretation of the algorithm, I suggest cs229.stanford.edu/notes/cs229-notes1.pdf. Not too heavy, so you don't get lost if you are a beginner. $\endgroup$ – alebu May 16 '17 at 15:47

If you consider logistic regression to be learning parameters $\beta_i$ for the function $$ f(\beta_0+\sum_{i=1}^{10}\beta_ix_i) $$ were $f$ transforms the linear sum to a probability range between 0 and 1. I think you may be failing to consider the $\beta_0$ term.

When x_test = [0,0..] therefore $x_i=0$ $$x_i = 0$$ $$y = f(\beta_0) = 0.43$$ In your case with the $\beta_0$ learning by the algorithm.

  • $\begingroup$ I really like your explanation. It make sense now. Thank you. :) $\endgroup$ – matox May 16 '17 at 11:13
  • $\begingroup$ No problem, glad it was useful $\endgroup$ – Harry Salmon May 16 '17 at 13:25

Your test sample that is all zeros (x=[0,0,0,..]) is not an empty test sample, but a sample with all features set at the mean, given that you state that all features were centered. Thus, the p = .43 is perfectly reasonable and reflects the predicted probability when all features are zero.

  • $\begingroup$ Thank you for you input, really appreciate it. I'm going to upvote it when I have enough of reputation. Thanks again :) $\endgroup$ – matox May 16 '17 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.