I want to use vector calculus to derive the gradient $\nabla_wJ(w)$ of the logistic regression cost function $J(w) = -\textbf{y}\cdot ln\textbf{ s} - (\mathbf{1} - \textbf{y}) \cdot ln( \mathbf{1} - \textbf{s})$, where $\textbf{s} = s(\textbf{Xw})$ is shorthand for the logistic function. I know my attempt is incorrect, but am not sure where I'm going wrong:

$$\nabla_wJ(w) = -\nabla_w[\textbf{y} \cdot ln\textbf{ s}] - \nabla_w[(\mathbf{1}-\textbf{y})\cdot ln(\mathbf{1} - \textbf{s})] = -([\nabla_w\textbf{y}] ln( s(\textbf{Xw}) + [\nabla_wln(s(\textbf{Xw}))]\textbf{y}) - ([\nabla_w(\mathbf{1} - \textbf{y})]ln(\mathbf{1} - s(\textbf{Xw})) + [\nabla_wln(\mathbf{1} - s(\textbf{Xw}))](\mathbf{1} - \textbf{y})) = \\ -[\nabla_wln(s(\textbf{Xw}))]\textbf{y} - [\nabla_wln(\mathbf{1} - s(\textbf{Xw}))](\mathbf{1} - \textbf{y}).$$ Now, since $\nabla_wln(s(\textbf{Xw})) = \mathbf{1} - s(\textbf{Xw})$, we can simplify and get:

$$-(\mathbf{1} - s(\textbf{Xw}))\textbf{y} + s(\textbf{Xw})(\mathbf{1} - \textbf{y}) = s(\textbf{Xw}) - \textbf{y}.$$ I know $\textbf{X}$ should be here somewhere, though, so something is wrong. I need to compute this gradient $\textbf{only}$ in terms of matrix-vector expressions, and so sums and extracting individual terms will not help. Thanks for any assistance in advance!

  • 1
    $\begingroup$ For a hint, look at that line starting "Now, since...". Note how when you take the derivative w.r.t. $\textbf{w}$, you don't seem to be taking what I think of as the "inner derivative" of $\textbf{Xw}$ - which would be getting you the $\textbf{X}$ you are looking for: $df(wx)/dw = xf'(wx)$, for example. $\endgroup$ – jbowman Mar 2 at 3:42
  • $\begingroup$ @jbowman This is helpful! I have a new doubt, which you'll see here: so now we have that $\nabla_wln(s(\mathbf{Xw})) = (1-s(\mathbf{Xw}))\mathbf{X}$, and $\nabla_wln(\mathbf{1} - s(\mathbf{Xw})) = -s(\mathbf{Xw})\mathbf{X}$, so we simplify and get $\nabla_wJ(w) = -(\mathbf{1}-s(\mathbf{Xw}))\mathbf{Xy} - (-s(\mathbf{Xw}))\mathbf{X}(\mathbf{1} - \mathbf{y}).$ This seems close, but there's a problem with the dimensions of $\mathbf{Xy}$ here... $\mathbf{X}$ is an (n x d) design matrix, and $\mathbf{y}$ is the (n x 1) vector of labels, and so the product isn't defined. Thanks for any more help! $\endgroup$ – Jake Mar 2 at 7:36
  • $\begingroup$ Have to be careful about derivatives and transposes when dealing with matrix algebra, no question about it! If you scroll down a bit from en.wikipedia.org/wiki/Matrix_calculus#Layout_conventions , you'll see a handy little table which may help keep things straight. The immediately preceding paragraph is a big help here too. Since your output $J(w)$ is a scalar, you are using "numerator layout", it seems to me. $\endgroup$ – jbowman Mar 2 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.