I've been following along with Andrew Ng's excellent course on Machine Learning (CS 229), and have been working on Problem Set 1.

I'm trying to fit a logistic regression model using Newton's Method. I have tried several approaches:

1) Calculating the components of $\mathbf{\theta} := \mathbf{\theta} - \mathbf{H}^{-1}\mathbf{\nabla}_{\mathbf{\theta}}\mathbf{\ell}$ element-by-element then solving;

2) Updating $\mathbf{\theta}$ using $\mathbf{(X^{T}WX)^{-1}X^{T}W\mathbf{z}}$ where $\mathbf{z}:=\mathbf{X}\theta+\mathbf{W^{-1}(y-p)}$. Described on slide 21 here.

3) Using the scikit's built-in package LogisticRegression to solve the system.

Number 1 gives me a singular Hessian. Number 2 gives a singluar weightings matrix $\mathbf{W}$ after 2 or 3 iterations. Number 3 converges to $\theta=[-1.0035,0.6191,1.0792]^{T}$. My data matrix $\mathbf{X}$ has 3 columns $[x_0,x_1,x_2]$, where $x_0$ is a column of ones. What am I doing wrong?

From a plot of the data (below), I don't believe the features are linearly dependent.

Plot of features.

Code for approach number 2 below. The dataframe 'data' has four columns $[x_0,x_1,x_2,y]$.

            import numpy as np
            import numpy.linalg as linalg
            import pandas as pd
            import csv as csv

            # Read in data to a pandas dataFrame. This file has 3 columns, 'x1', 'x2' and 'y'
            data = pd.read_csv(r'.\logistic_xy.csv')

            # Convert to a numpy array
            data = np.array(data)

            # Define data matricies

            X = np.array(data[0::,0:3])
            y = np.array(data[0::,3])
            y = y.reshape((99,1))
            W = np.zeros([99,99],dtype=float)
            p = np.zeros([99,1],dtype=float)
            z = np.zeros([99,1], dtype=float)

            # Define hypothesis function
            def hTheta(x):
                g = np.exp(x)/(1+np.exp(x))
                return g

            # Define convergence variable and initialize theta

            theta = np.zeros([3,1],dtype=float)
            thetaNew = theta
            conv = 1

            while conv > 0.01:
                for i in xrange(0,int(data[0::,0].sum())):
                    theta = thetaNew
                    x = np.dot(np.transpose(theta),data[i,0:3])[0]
                    h = hTheta(x)
                    W[i,i] = h*(1-h)
                    p[i] = h

                z = np.add(np.dot(X,theta),np.dot(np.linalg.inv(W),np.subtract(y,p)))
                A = np.linalg.inv(np.dot(X.transpose(),np.dot(W,X)))
                B = np.dot(X.transpose(),np.dot(W,z))

                thetaNew = np.dot(A,B)
                conv = np.linalg.norm(np.subtract(thetaNew,theta))
  • 1
    $\begingroup$ Do you want to add the self-study tag? It seems appropriate as you are trying to answer a question given in a course (even though it is an online course) $\endgroup$ – Ferdi Oct 22 '16 at 16:32
  • $\begingroup$ Are your covariates linearly independent? $\endgroup$ – Adrian Oct 22 '16 at 16:47
  • $\begingroup$ I think so, given that the pre-built package solved with no issues. Do you have a suggestion for how I could check this? The input data are given here: cs229.stanford.edu/ps/ps1/logistic_x.txt cs229.stanford.edu/ps/ps1/logistic_y.txt $\endgroup$ – Eric Oct 22 '16 at 16:48
  • $\begingroup$ I believe it must be the problem of initial value of Newton's Method. If given bad initial values, Newton's Method won't converge. In my case, after several iterations, bad initial values make coefficients surprisingly large and mess up everything. $\endgroup$ – Naomi Feb 3 '18 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.