# Logistic regression and singular Hessian

I've been following along with Andrew Ng's excellent course on Machine Learning (CS 229), and have been working on Problem Set 1.

I'm trying to fit a logistic regression model using Newton's Method. I have tried several approaches:

1) Calculating the components of $\mathbf{\theta} := \mathbf{\theta} - \mathbf{H}^{-1}\mathbf{\nabla}_{\mathbf{\theta}}\mathbf{\ell}$ element-by-element then solving;

2) Updating $\mathbf{\theta}$ using $\mathbf{(X^{T}WX)^{-1}X^{T}W\mathbf{z}}$ where $\mathbf{z}:=\mathbf{X}\theta+\mathbf{W^{-1}(y-p)}$. Described on slide 21 here.

3) Using the scikit's built-in package LogisticRegression to solve the system.

Number 1 gives me a singular Hessian. Number 2 gives a singluar weightings matrix $\mathbf{W}$ after 2 or 3 iterations. Number 3 converges to $\theta=[-1.0035,0.6191,1.0792]^{T}$. My data matrix $\mathbf{X}$ has 3 columns $[x_0,x_1,x_2]$, where $x_0$ is a column of ones. What am I doing wrong?

From a plot of the data (below), I don't believe the features are linearly dependent.

Code for approach number 2 below. The dataframe 'data' has four columns $[x_0,x_1,x_2,y]$.

            import numpy as np
import numpy.linalg as linalg
import pandas as pd
import csv as csv

# Read in data to a pandas dataFrame. This file has 3 columns, 'x1', 'x2' and 'y'

# Convert to a numpy array
data = np.array(data)

# Define data matricies

X = np.array(data[0::,0:3])
y = np.array(data[0::,3])
y = y.reshape((99,1))
W = np.zeros([99,99],dtype=float)
p = np.zeros([99,1],dtype=float)
z = np.zeros([99,1], dtype=float)

# Define hypothesis function
def hTheta(x):
g = np.exp(x)/(1+np.exp(x))
return g

# Define convergence variable and initialize theta

theta = np.zeros([3,1],dtype=float)
thetaNew = theta
conv = 1

while conv > 0.01:
for i in xrange(0,int(data[0::,0].sum())):
theta = thetaNew
x = np.dot(np.transpose(theta),data[i,0:3])[0]
h = hTheta(x)
W[i,i] = h*(1-h)
p[i] = h