2
$\begingroup$

Suppose we have a neural net with only the input and output layers, sigmoid activations and quadratic cost $C = {(y-a)^2 \over 2}$ (one training example) where $y$ is the desired output, $z=\sum_j w_jx_j$, $a = \sigma(z)$, $x_j$ is the $j$th input and $w_j$ is the $j$th weight. Now, the partial derivative of the cost function with respect to a given weight is $${\partial C \over \partial w_j}=x_j(a-y){\sigma}^{'}(z)$$ In particular, the learning rate depends on the input, the distance from the desired output and the saturation of the sigmoid function.

If we switch to a cross-entropy cost function (still for one training example) $$C = -\left[y \ln a + (1-y ) \ln (1-a) \right]$$ we get the partial derivative with respect to a given weight $${\partial C \over \partial w_j} = x_j(\sigma(z)-y)$$ So, the learning rate doesn't depend on the saturation of the sigmoid.

Now, the question. Why can't we get rid of the $x_j$ part of ${\partial C \over \partial w_j}$ by choosing some clever cost function?

My intuition is that for backpropagation to work, we must have $C = C(a)$ and so the cost for a training input is also a function of $w$'s and $b$ (weights and bias). If we take a partial derivative with respect to $w_j$, we will get the $x_j$ term. This is very far from formal. Do you have any suggestions for a more formal argument?

$\endgroup$
  • 1
    $\begingroup$ The optimization is based on $w_j\leftarrow w_j-\alpha \frac{\partial C}{\partial w_j}$, and far away from the optimum point, the job of $\frac{\partial C}{\partial w_j}$ is to give you the right sign (i.e. direction) to move in. So you could use $\frac{\partial C}{\partial w_j}=\mbox{sgn}(x)(\sigma(z)-y)$, where you're just accounting for the sign of $x$. In particular, if you normalize your $x_i$ before optimizing, to live in say, $[-1,1]$, then this gives you an aggressive gradient descent. If you scale $\alpha$ with time, then this might even converge. $\endgroup$ – Alex R. Oct 23 '17 at 18:03
1
$\begingroup$

According to the chapter-2 from the book of Michael Nielsen, two assumptions are required regarding the form of the cost function for backpropagation to work properly. One of the assumptions is to define the cost as a function of the output activations.

Now, whatever activation function (i.e., sigmoid, softmax, linear) you are using in your neural network, whenever you take the partial derivative of the cost function with respect to weight, the derivative of the weighted input will always produce an additional $x_{j}$ due to the application of the chain rule. It is not possible to get rid of this additional $x_{j}$ due to the constraint that cost has to be a function of the output activations.

N.B.: I don't think there is a very formal argument required to approach this question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.