3
$\begingroup$

I'm a software engineer, and I have just started a Udacity's nanodegree of deep learning.

I have also worked my way through Stanford professor Andrew Ng's online course on machine learning and now I'm comparing.

I have a big doubt about Gradient Descent with sigmoid function because on Andrew Ng's course it is different from the one I see on Udacity's nanodegree.

From Andrew Ng's course, gradient descent is (First formula): enter image description here

But, from Udacity's nanodegree is (Second formula):

enter image description here

Note: first picture is from this video, and second picture is for this other video.

But in this CS229 course notes from Andrew Ng's, on page 18, I have found the demonstration from Andrew Ng's gradient ascent formula. I only add it here as a demostration: enter image description here

Note: the formula above is for Gradient Ascent.

I'm not sure if I have understood everything, but in this derivative I see have the derivative from f function disappears (f function is the sigmoid function).

But in Udacity's nanodegree they continue using the sigmoid's derivative in their gradient descent.

The difference between first formula and second formula is the derivative term.

Are the two formula equivalents?

$\endgroup$
  • $\begingroup$ I don't think Andrew has a course with an invalid gradient descent. I think there is something that I don't understand with maths. $\endgroup$ – VansFannel Feb 13 '17 at 18:41
  • $\begingroup$ I have found another example saying that the First formula is the correct one: ataspinar.com/2016/03/28/… $\endgroup$ – VansFannel Feb 14 '17 at 13:36
4
$\begingroup$

Both formulas are correct. Here is how you can get formula 2, by minimize the sum of squared errors. $$ \sum^n_i(y^i-\hat{y^i})^2 $$ Dividing it by $n$ gives you MSE, and by $2$ gives you SSE used in the formula 2. Since you are going to minimize this expression with partial derivation technique, choosing $2$ makes the derivation look nice.

Now you use a linear model where $\hat{y} = f(h)$ and $h(x) = \theta{x}$, you get, (I omit the transpose symbol for $\theta$ in $\theta^T{x}$) $$ \frac{1}{2}\sum^n_i(y^i-{f(\theta{x^i})})^2 $$ When you compute its partial derivative over $\theta$ for the additive term, you have, $$ \frac{\partial}{\partial\theta_j}(\frac{1}{2}(y-{f(\theta{x})})^2) = -(y-\hat{y}) f'(\theta{x}) x_j \quad (1) $$

This is the formula 2 you gave. I don't give the detailed steps, but it is quite straightforward.

Yes, $f(h)$ is the activation function, and you do have the factor $f'(h)$ in the derivative expression as shown above. It disappears if it equals 1, i.e., $f(h) = h + c$, where $c$ is invariable w.r.t. $h$.

For example, if $h_\theta(x) = \theta{x}$, (i.e., $\sum^n_i{\theta_i{x_i}}$), and the prediction model is linear where $f(x) = \theta{x}$, too, then you have $f(h) = h$ and $f'(h) = 1$.

For another example, if $h_\theta(x) = \theta{x}$, while the prediction model is sigmoid where $f(h) = \frac{1}{1+e^-h}$, then $f'(h) = f(h)(1-f(h))$. This is why in book Artificial Intelligence: A Modern Approach, the derivative of logistic regression is: $$ \frac{\partial}{\partial\theta_j}(\frac{1}{2}(y-{f(\theta{x})})^2) = -(y-\hat{y}) \hat{y}(1-\hat{y})x_j \quad (2) $$ On the other hand, the formula 1, although looking like a similar form, is deduced via a different approach. It is based on the maximum likelihood (or equivalently minimum negative log-likelihood) by multiplying the output probability function over all the samples and then taking its negative logarithm, as given below, $$ \sum^n_i-\log{P(y^i|x^i; \theta)} $$

In a logistic regression problem, when the outputs are 0 and 1, then each additive term becomes,

$$ −logP(y^i|x^i;\theta) = -(y^i\log{h_\theta(x^i)} + (1-y^i)\log(1-h_\theta(x^i))) $$

The formula 1 is the derivative of it (and its sum) when $h_\theta(x) = \frac{1}{1+e^{-\theta{x}}}$, as below, $$ \frac{\partial}{\partial\theta_{j}}(−logP(y|x;\theta)) = -(y - \hat{y})x_j \quad (3) $$ The derivation details are well given in other post.

You can compare it with equation (1) and (2). Yes, equation (3) does not have the $f'(h)$ factor as equation (1), since that is not part of its deduction process at all. Equation (2) has additional factors $\hat{y}(1-\hat{y})$ compared to equation (3). Since $\hat{y}$ as probability is within the range of (0, 1), you have $\hat{y}(1-\hat{y}) < 1$, that means equation (2) brings you a gradient of smaller absolute value, hence a slower convergence speed in gradient descent than equation (3).

Note the sum squared errors (SSE) essentially a special case of maximum likelihood when we consider the prediction of $\hat{y}$ is actually the mean of a conditional normal distribution. That is, $$ p(y | x) = N(y;\hat{y},I). $$ If you want to get more in depth knowledge in this area, I would suggest the Deep Learning book by Ian Goodfellow, et al.

$\endgroup$
4
$\begingroup$

So, I think you are mixing up minimizing your loss function, versus maximizing your log likelihood, but also, (since both are equivalent), the equations you have written are actually the same. Let's assume there is only one sample, $i=1$ to keep things simple.

The first equation shows the minimization of loss update equation: $$\theta_j = \theta_j -\alpha \Big(h_{\theta}(x^1) - y^1\Big)x_j^1$$ Here, the gradient of the loss is given by:

$$ \Big(h_{\theta}(x^1) - y^1\Big)x_j^1 $$

However, the third equation you have written:

$$\frac{\delta l(\theta)}{\delta \theta_j } = \Big(y^1 - h_{\theta}(x^1)\Big)x_j^1$$

is not the gradient with respect to the loss, but the gradient with respect to the log likelihood!

This is why you have a discrepancy in your signs.

Now look back at your first equation, and open the gradient with the negative sign - you will then be maximizing the log likelihood, which is the same as minimizing the loss. Therefore they are all equivalent. Hope that helped!

$\endgroup$
  • $\begingroup$ Thanks. I know the problem with the sings. But on the first equation there is f prime function that there isn't on the second one. That is what I don't understand. $\endgroup$ – VansFannel Feb 13 '17 at 22:19
  • $\begingroup$ @VansFannel If you listen to the udacity lecture, he says that the derivative of $f$ is the derivative of the activation function. However in Andrew Ng's lectures, I do not believe he is using an activation function at all. $\endgroup$ – Tarin Ziyaee Feb 13 '17 at 22:38
  • $\begingroup$ Thanks a lot for your answer but I haven't understood anything. I think you are right when you say that both functions, Coursera's and Udacity's function, are the same but I don't understand why. Do you know where I can go in depth these topics (gradient descent, log likelihood)? My problem is that I don't understand why the equations I have written are actually the same (one don't have the derivative and the other one has it).Thanks. $\endgroup$ – VansFannel Feb 14 '17 at 7:57
  • $\begingroup$ I think, I have found how to get the First formula here: math.stackexchange.com/a/477261/193243. But I don't know why they are using the Second formula. $\endgroup$ – VansFannel Feb 14 '17 at 13:29
  • $\begingroup$ I'm still trying to understand the different between two formulas. I think in the First formula activation function (lineal function or sigmoid function is h), but in the Second formula the activation function seems to be f. But, if f is the activation function, what is h? $\endgroup$ – VansFannel Feb 15 '17 at 18:27
0
$\begingroup$

The two formulas are for different loss functions, one of which is generally much better than the other when performing logistic regression.

The first formula is used for calculating the output node deltas when using binary cross entropy loss and a sigmoid activation function for the output nodes. This is the formula I would use for logistic regression, and it also happens to be the formula for linear regression despite a different derivation.

The second formula is a more general expression for using mean squared error loss when you have an activation function for your output node. In the case of linear regression, f′(h) = 1, because there is no activation function (it's the identity function), so you're left with the first formula. In the case of logistic regression, f′(h) = f(h) * (1 - f(h)). I would not recommend using mean squared error loss for logistic regression, as it's very slow. Binary cross entropy is a much better loss function to use with logistic regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.