Consider the prediction of a binary value $G$ on the basis of a quantitative predictor $X_1$. I use a linear logistic regression (without intercept) for predictions. I want to know if I have formed the data generating model, max likelihood objective function, and solved the optimization problem using Newton's iterative method appropriately - basically showing the expression for the successive approximations of the logistic regression weight. I want to ensure that my solution is expressed in a way such that the expression can be directly obtained from $G$ and $X_1$ using basic operations like summation, multiplication, logarithm, and exponential. Here is my working so far:

The data generating model is given by the following two variables :

$$P(G = 1 | X_ 1, \: \beta) = \dfrac{1}{1 + e^{-X_1\beta}} $$ $$P(G = 0 | X_ 1, \: \beta) = 1 - \dfrac{1}{1 + e^{-X_1\beta}} $$

Both can be written as:

$$P(G = g | X_1, \: \beta) = \left( \dfrac{1}{1+e^{-X_1\beta}}\right)^{g}\left( 1 - \dfrac{1}{1 + e^{-X_1\beta}}\right)^{1-g}$$

Let there be $N$ samples in the data, and let $\Lambda(\beta)$ be the likelihood function. $$\Lambda(\beta) = \prod^N_{i=1} P(g_i | x_{1i}, \: \beta)$$

where $y_i$ is the probability calculated using $x_{1i}$ and the coefficient $\beta$.

Now, let $\lambda(\beta) = log(\Lambda(\beta))$ and $H(x_{1i}) = \dfrac{1}{1 + e^{-X_{1i}\beta}} $. The objective function to optimize is:

$$\lambda(\beta) = \sum^N_{i=1} \left[g_i log(H(x_{1i})) + (1 - g_i)log(1 - H(x_{1i}))\right]$$

Newton’s algorithm uses $\lambda(\beta)$ and the value of $\beta$ at the $n$-th iteration to calculate the value of $\beta$ at the $(n+1)$-th iteration. The algorithm is as follows : $$\beta_{n+1} = \beta_n - \dfrac{\lambda'(\beta)}{\lambda''(\beta)}$$

We stop when $\lambda(\beta_{n+1}) - \lambda(\beta_{n})$. Note that $\dfrac{d(H)}{d\beta} = H \times (1-H) \times X$.

$$\therefore \dfrac{d(\lambda(\beta))}{d\beta} = \sum^N_{i=1}\left[\dfrac{g_i}{H(x_{1i})}H(x_{1i})(1-H(x_{1i}))x_{1i} - \dfrac{1-g_i}{1-H(x_{1i})}H(x_{1i})(1-H(x_{1i}))x_{1i}\right]$$

$$\therefore \dfrac{d(\lambda(\beta))}{d\beta} = \sum^N_{i=1}\left[(g_i - H(x_{1i}))x_{1i}\right]$$

$$\therefore \dfrac{d^2(\lambda(\beta))}{d\beta^2} = \sum^N_{i=1}\left[H(x_{1i})(H(x_{1i})-1)x_{1i}^2\right]$$

So, the Newton’s algorithm can be written as:

$$\beta_{n+1} = \beta_n - \dfrac{\sum^N_{i=1}\left[(g_i - H(x_{1i}))x_{1i}\right]}{\sum^N_{i=1}\left[H(x_{1i})(H(x_{1i})-1)x_{1i}^2\right]} $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.