I am testing Logistic Regression with stochastic gradient using sklearn.linear_model.SGDClassifier. I have 2D independent variables and corresponding labels as below:

X = array([[-2.58733628,  2.26126322],
          [ 1.97831473,  2.03510032],
          [ 2.48324069, -2.17901384], ... ])
Y = array([[ 1.], [ 1.], [-1.], [-1.], ...])

Here my objective function is $\sum \log( 1 + \exp(-y^{(i)}(wx^{(i)}+ w_{0}))) + \lambda||w||_2^2$, and I first tested with no regularization term.

mySGDlr = linear_model.SGDClassifier(loss = 'log')
print mySGDlr.score(X, ravel(Y))

1) Then, the score is 1, which is not possible. Could you point me where is the wrong part in my code? Also, how can I check the optimized $w, w_0$ after having fitted the model?

2) I read the documentation of SGDClassifier and LogisticRegression function, and it seems that the main difference is that SDGClassifier uses SGD, and LogisticRegression uses some other fancy solvers. Is there any other importance differences?

  • $\begingroup$ The question about code is off-topic, but the rest of the question is on-topic, imo. As you mentioned you tested without regularization, are you sure your data are not linearly separable? $\endgroup$ – Firebug Oct 16 '16 at 2:45
  • $\begingroup$ I am sure that my data is not linearly separable. By the way, if I have specific questions regarding codes, such as linear_model.SGDClassifier, where I should post questions? Sorry for my ignorance! $\endgroup$ – nClew Oct 16 '16 at 3:26
  • $\begingroup$ The only reason to use SGD for logistic regression is if your data are too large to fit into memory. The logistic regression objective can be easily solved using some pretty simple matrix arithmetic. $\endgroup$ – Sycorax Oct 16 '16 at 3:34
  • $\begingroup$ @Sycorax I would be interested in seeing that "simple matrix arithmetic." AFAIK it's not possible, due to the irreducible nonlinearity of the objective function. $\endgroup$ – whuber Oct 16 '16 at 17:46
  • 1
    $\begingroup$ @whuber Sorry for the lack of clarity -- I was referring to iteratively re-weighted least squares/Newton's method -- which are both pretty simple: the objective is convex, so it's a descent algorithm. SGD is not necessarily a descent algorithm, for example, which I think makes its application here more challenging. $\endgroup$ – Sycorax Oct 16 '16 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.