# Logistic Regression with Newton's method

Consider the prediction of a binary value $$G$$ on the basis of a quantitative predictor $$X_1$$. I use a linear logistic regression (without intercept) for predictions. I want to know if I have formed the data generating model, max likelihood objective function, and solved the optimization problem using Newton's iterative method appropriately - basically showing the expression for the successive approximations of the logistic regression weight. I want to ensure that my solution is expressed in a way such that the expression can be directly obtained from $$G$$ and $$X_1$$ using basic operations like summation, multiplication, logarithm, and exponential. Here is my working so far:

The data generating model is given by the following two variables :

$$P(G = 1 | X_ 1, \: \beta) = \dfrac{1}{1 + e^{-X_1\beta}}$$ $$P(G = 0 | X_ 1, \: \beta) = 1 - \dfrac{1}{1 + e^{-X_1\beta}}$$

Both can be written as:

$$P(G = g | X_1, \: \beta) = \left( \dfrac{1}{1+e^{-X_1\beta}}\right)^{g}\left( 1 - \dfrac{1}{1 + e^{-X_1\beta}}\right)^{1-g}$$

Let there be $$N$$ samples in the data, and let $$\Lambda(\beta)$$ be the likelihood function. $$\Lambda(\beta) = \prod^N_{i=1} P(g_i | x_{1i}, \: \beta)$$

where $$y_i$$ is the probability calculated using $$x_{1i}$$ and the coefficient $$\beta$$.

Now, let $$\lambda(\beta) = log(\Lambda(\beta))$$ and $$H(x_{1i}) = \dfrac{1}{1 + e^{-X_{1i}\beta}}$$. The objective function to optimize is:

$$\lambda(\beta) = \sum^N_{i=1} \left[g_i log(H(x_{1i})) + (1 - g_i)log(1 - H(x_{1i}))\right]$$

Newton’s algorithm uses $$\lambda(\beta)$$ and the value of $$\beta$$ at the $$n$$-th iteration to calculate the value of $$\beta$$ at the $$(n+1)$$-th iteration. The algorithm is as follows : $$\beta_{n+1} = \beta_n - \dfrac{\lambda'(\beta)}{\lambda''(\beta)}$$

We stop when $$\lambda(\beta_{n+1}) - \lambda(\beta_{n})$$. Note that $$\dfrac{d(H)}{d\beta} = H \times (1-H) \times X$$.

$$\therefore \dfrac{d(\lambda(\beta))}{d\beta} = \sum^N_{i=1}\left[\dfrac{g_i}{H(x_{1i})}H(x_{1i})(1-H(x_{1i}))x_{1i} - \dfrac{1-g_i}{1-H(x_{1i})}H(x_{1i})(1-H(x_{1i}))x_{1i}\right]$$

$$\therefore \dfrac{d(\lambda(\beta))}{d\beta} = \sum^N_{i=1}\left[(g_i - H(x_{1i}))x_{1i}\right]$$

$$\therefore \dfrac{d^2(\lambda(\beta))}{d\beta^2} = \sum^N_{i=1}\left[H(x_{1i})(H(x_{1i})-1)x_{1i}^2\right]$$

So, the Newton’s algorithm can be written as:

$$\beta_{n+1} = \beta_n - \dfrac{\sum^N_{i=1}\left[(g_i - H(x_{1i}))x_{1i}\right]}{\sum^N_{i=1}\left[H(x_{1i})(H(x_{1i})-1)x_{1i}^2\right]}$$