# gradient descent for logistic regression

I'm implementing (for learning purposes) a logistic regression model.

I've followed this guide.

Now, the author is taking the derivative of $$l$$, the cost function with respect to some $$\beta_j$$:

$$\frac{\partial l}{\partial \beta_j} =\sum_{i = 1}^m y_i x_{ij} - \sum_{i = 1}^m \frac{e^{\beta_0 + \beta \cdot x_i}}{1 + e^{\beta_0 + \beta \cdot x_i}} \cdot x_{ij} \\ = \sum_{i=1}^m x_{ij} (y_i - p(x_i;\beta_0, \beta))$$

my question is regarding $$\beta_0$$. Basically the derivative is different for $$\beta_0$$.

Is that reasonable, for simplicity, to ommit $$\beta_0$$ and instead extending $$x_i$$ to a larger dimension where the first term is a constant of $$1$$?

• Yes, it's reasonable to omit $\beta_0$, but it's common to just abuse notation slightly instead. That is, we write the linear predictor as $\beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_px_p$, but think of $x$ as a $(p+1)$-vector with first entry 1. So we're thinking of $\beta_0\times 1$ as the first term, but writing $\beta_0$. – Thomas Lumley Jan 25 at 0:27

No, the partial derivative is not different for $$\beta_0$$. In your formulation, just rewrite the linear predictor as $$\mathbf{x}^\top \boldsymbol{\beta}$$ instead of $$\beta_0 + \beta x_i$$, since $$x_{i0}=1$$ for every record and therefore $$\beta_0$$ can just be treated as another coefficient. In other words, $$\beta_0 + \beta x_i$$, is really $$\beta_0 x_{i0} + \beta_1 x_{i1}$$, which is equal to vector multiplication via $$\mathbf{x}^\top \boldsymbol{\beta}$$. $$\beta_0$$ is not hanging alone without its own $$x$$. When solving for $$\beta_0$$ just include it in the same derivative computations for the score vector and Hessian matrix, i.e., $$\partial l / \partial \beta_j$$, and $$\frac{\partial l^2}{ \partial \beta_j \partial \beta_k}$$.