gradient descent for logistic regression

Question

I'm implementing (for learning purposes) a logistic regression model.

I've followed this guide.

Now, the author is taking the derivative of $l$, the cost function with respect to some $\beta_j$:

$$\frac{\partial l}{\partial \beta_j} =\sum_{i = 1}^m y_i x_{ij} - \sum_{i = 1}^m \frac{e^{\beta_0 + \beta \cdot x_i}}{1 + e^{\beta_0 + \beta \cdot x_i}} \cdot x_{ij} \\ = \sum_{i=1}^m x_{ij} (y_i - p(x_i;\beta_0, \beta))$$

my question is regarding $\beta_0$. Basically the derivative is different for $\beta_0$.

Is that reasonable, for simplicity, to ommit $\beta_0$ and instead extending $x_i$ to a larger dimension where the first term is a constant of $1$?

Answer 1

No, the partial derivative is not different for $\beta_0$. In your formulation, just rewrite the linear predictor as $\mathbf{x}^\top \boldsymbol{\beta}$ instead of $\beta_0 + \beta x_i$, since $x_{i0}=1$ for every record and therefore $\beta_0$ can just be treated as another coefficient. In other words, $\beta_0 + \beta x_i$, is really $\beta_0 x_{i0} + \beta_1 x_{i1}$, which is equal to vector multiplication via $\mathbf{x}^\top \boldsymbol{\beta}$. $\beta_0$ is not hanging alone without its own $x$. When solving for $\beta_0$ just include it in the same derivative computations for the score vector and Hessian matrix, i.e., $\partial l / \partial \beta_j$, and $\frac{\partial l^2}{ \partial \beta_j \partial \beta_k}$.