I am making a naive logistic regression implementation to better understand how it works. I am using a very simple dataset provided by Andrew Ng's course on Coursera (pastebin link with the rest of the code).

Weirdly enough, the model seems to be getting progressively worst instead of better. For example, in the last training it did, 74% -> 73% -> 71% the it stagnates until the training is complete.

Output example:

Logistic regression results

I tried:

  • Changing the learning constant alpha
  • Using mini-batch and stochastic gradient descent (the latter obviously didn't work because the set contains only a hundred examples)
  • Using a different dataset (the other one provided with the course)
  • My initial thetas are randomly assigned between 0 and 1 instead of 0 (it did a bit better but that clearly isn't the problem)
  • Adding polynomials (for those who checked the code) doesn't change anything

My guesses (What I'm investigating right now):

  • My cost function is broken (it give negative values as you can see)
  • The is an inversion somewhere in the code that increases the error instead of lowering it.

My code:

Sigmoid function:

    def sigmoid(x):
        return 1 / (1+np.exp(-x))

Cost function:

    def cost_function(X, y, theta):
        num_examples = np.shape(X)[0]
        h = sigmoid(np.dot(X, np.transpose(theta)))
        return ((ALPHA / num_examples) * sum((np.dot(np.transpose(y), np.log(h))) - np.dot(np.transpose(1 - y), np.log(1 - h))))[0]


    def batch_gradient(X, y, theta):
        num_examples = np.shape(X)[0]
        for i in range(0, NUM_ITERS):
            h = sigmoid(np.dot(X, np.transpose(theta)))
            theta = theta - ALPHA/num_examples * np.transpose((np.dot(np.transpose(X),(h - y))))
            if i % 10000 == 0:
                print(str(i) + '/' + str(NUM_ITERS))
                print('Cost function value: {:.10f}'.format(cost_function(X, y, theta)))
                test(X, y, theta)
        return theta

Full source: http://pastebin.com/2ep2xK4x

First dataset: http://pastebin.com/LhG2c8vb

Second dataset: http://pastebin.com/AudGzYmX (Very interestingly, this one doesn't work at all where the predictor always predict true)

Second data set

Any clue to help me dig in the right direction would be great. I'm still actively looking into the issue so I'll update the post if I find anything.

Edit: Fixed the title

Edit2: Added observation concerning the second dataset.

Edit3: Fixed format and added informations concerning the cost function.

Edit4: Tried more things, didn't give any interesting improvement.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.