1
$\begingroup$

I'm implementing (for learning purposes) a logistic regression model.

I've followed this guide.

Now, the author is taking the derivative of $l$, the cost function with respect to some $\beta_j$:

$$\frac{\partial l}{\partial \beta_j} =\sum_{i = 1}^m y_i x_{ij} - \sum_{i = 1}^m \frac{e^{\beta_0 + \beta \cdot x_i}}{1 + e^{\beta_0 + \beta \cdot x_i}} \cdot x_{ij} \\ = \sum_{i=1}^m x_{ij} (y_i - p(x_i;\beta_0, \beta))$$

my question is regarding $\beta_0$. Basically the derivative is different for $\beta_0$.

Is that reasonable, for simplicity, to ommit $\beta_0$ and instead extending $x_i$ to a larger dimension where the first term is a constant of $1$?

$\endgroup$
1
  • $\begingroup$ Yes, it's reasonable to omit $\beta_0$, but it's common to just abuse notation slightly instead. That is, we write the linear predictor as $\beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_px_p$, but think of $x$ as a $(p+1)$-vector with first entry 1. So we're thinking of $\beta_0\times 1$ as the first term, but writing $\beta_0$. $\endgroup$ – Thomas Lumley Jan 25 at 0:27
2
$\begingroup$

No, the partial derivative is not different for $\beta_0$. In your formulation, just rewrite the linear predictor as $\mathbf{x}^\top \boldsymbol{\beta}$ instead of $\beta_0 + \beta x_i$, since $x_{i0}=1$ for every record and therefore $\beta_0$ can just be treated as another coefficient. In other words, $\beta_0 + \beta x_i$, is really $\beta_0 x_{i0} + \beta_1 x_{i1}$, which is equal to vector multiplication via $\mathbf{x}^\top \boldsymbol{\beta}$. $\beta_0$ is not hanging alone without its own $x$. When solving for $\beta_0$ just include it in the same derivative computations for the score vector and Hessian matrix, i.e., $\partial l / \partial \beta_j$, and $\frac{\partial l^2}{ \partial \beta_j \partial \beta_k}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.