Recently, I implemented the LR algorithm in Python. The main part of the code is as following(I didn't use mini batch in my code. Instead, I use the whole batch to compute gradients every time):

class Logistic():
def __init__(self):
    self.w = None
    self.lr = 10.
def train(self, xs, ys):
    m, n = xs.shape
    ones = np.ones([m, 1])
    xs = np.hstack([xs, ones])
    ys = np.expand_dims(ys, -1)
    self.w = np.ones([n+1, 1], dtype=np.float64) * 1.0
    epochs = 100
    for epoch in xrange(epochs):
        y_ = self.sigmoid(-np.dot(xs, self.w))
        # loss = -1.0/m * np.sum(ys * np.log(y_) + (1 - ys) * np.log(1 - y_))
        tmp1 = np.sum(np.log(y_[np.where(ys==1)]))
        tmp2 = np.sum(np.log(1 - y_[np.where(ys==0)]))
        loss = - (tmp1 + tmp2) / m
        print("epoch: %d, loss: %f" % (epoch, loss))
        print("y_: %f, %f" % (np.min(y_), np.max(y_)))
        grad =  np.sum((y_ - ys) * xs, axis=0) / m
        self.w -= self.lr * np.expand_dims(grad, -1)
        print("grad: %f, %f" % (np.min(grad), np.max(grad)))
        print("w: %f, %f" % (np.min(self.w), np.max(self.w)))
        print ""

The dataset I used is MNIST. I marked all digits 0 as class 0, and all other digits as class 1. Then I get this binary classification problem. I test my algorithm with many different learning rate, from 1e-6 to 10, and it turns out all of them produces good results(about 98% accuracy on test set). As far as I know, if the learning rate is to big, LR will not converge. But here although I used very big learning rate, the algorithm still converge to about 98% accuracy. Is there an explanation for this?

  • $\begingroup$ What is the proportion of $0$'s in your dataset (I presume it is 10 % in which accuracy is misleading, since a dummy classifier can make 90 % by assigning each given sample to class non-zero)? What is your classification threshold? Can you provide a confusion matrix also? $\endgroup$ – gunes Apr 2 at 11:34
  • $\begingroup$ A good debugging strategy is to compare your result to one obtained with a tool that's known to work correctly. When you use another logistic regression method, what coefficients do you obtain? Are they the same or different? $\endgroup$ – Sycorax Apr 2 at 12:57
  • $\begingroup$ I now think that the reason may be that I didn't include L1 or L2 regularization in the loss function. So when I used very big learning rate, the weights produced by the algorithm became very big(I checked this by printing weights out). But the prediction it produced is still correct. $\endgroup$ – Swi Jason Apr 3 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.