# Logistic regression getting worse, what are the common causes?

I am making a naive logistic regression implementation to better understand how it works. I am using a very simple dataset provided by Andrew Ng's course on Coursera (pastebin link with the rest of the code).

Weirdly enough, the model seems to be getting progressively worst instead of better. For example, in the last training it did, 74% -> 73% -> 71% the it stagnates until the training is complete.

Output example:

I tried:

• Changing the learning constant alpha
• Using mini-batch and stochastic gradient descent (the latter obviously didn't work because the set contains only a hundred examples)
• Using a different dataset (the other one provided with the course)
• My initial thetas are randomly assigned between 0 and 1 instead of 0 (it did a bit better but that clearly isn't the problem)
• Adding polynomials (for those who checked the code) doesn't change anything

My guesses (What I'm investigating right now):

• My cost function is broken (it give negative values as you can see)
• The is an inversion somewhere in the code that increases the error instead of lowering it.

My code:

Sigmoid function:

    def sigmoid(x):
return 1 / (1+np.exp(-x))


Cost function:

    def cost_function(X, y, theta):
num_examples = np.shape(X)[0]
h = sigmoid(np.dot(X, np.transpose(theta)))
return ((ALPHA / num_examples) * sum((np.dot(np.transpose(y), np.log(h))) - np.dot(np.transpose(1 - y), np.log(1 - h))))[0]


    def batch_gradient(X, y, theta):
num_examples = np.shape(X)[0]
for i in range(0, NUM_ITERS):
h = sigmoid(np.dot(X, np.transpose(theta)))
theta = theta - ALPHA/num_examples * np.transpose((np.dot(np.transpose(X),(h - y))))
if i % 10000 == 0:
print(str(i) + '/' + str(NUM_ITERS))
print('Cost function value: {:.10f}'.format(cost_function(X, y, theta)))
test(X, y, theta)
return theta


Full source: http://pastebin.com/2ep2xK4x

First dataset: http://pastebin.com/LhG2c8vb

Second dataset: http://pastebin.com/AudGzYmX (Very interestingly, this one doesn't work at all where the predictor always predict true)

Any clue to help me dig in the right direction would be great. I'm still actively looking into the issue so I'll update the post if I find anything.

Edit: Fixed the title

Edit2: Added observation concerning the second dataset.

Edit3: Fixed format and added informations concerning the cost function.

Edit4: Tried more things, didn't give any interesting improvement.