0
$\begingroup$

I've just started reading about neural networks from neural networks and deep learning and came across this section about the quadratic cost function vs taking accuracy directly to improve weights.

Why introduce the quadratic cost? After all, aren't we primarily interested in the number of images correctly classified by the network? Why not try to maximize that number directly, rather than minimizing a proxy measure like the quadratic cost? The problem with that is that the number of images correctly classified is not a smooth function of the weights and biases in the network. For the most part, making small changes to the weights and biases won't cause any change at all in the number of training images classified correctly. That makes it difficult to figure out how to change the weights and biases to get improved performance. If we instead use a smooth cost function like the quadratic cost it turns out to be easy to figure out how to make small changes in the weights and biases so as to get an improvement in the cost. That's why we focus first on minimizing the quadratic cost, and only after that will we examine the classification accuracy.

Could someone help me figure out how the quadratic cost function is "smooth" compared to accuracy?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ There are two points. (A) It is continuous in that a small change in the weights and biases leads to a small change in the cost; by contrast a classification accuracy measure is unchanged for many small changes but jumps discretely for some small changes. (B) It is differentiable so the effect of a certain small change is close to the effect of repeating that small change; by contrast an absolute value cost function can see kinks in the effects of changes $\endgroup$
    – Henry
    May 16, 2017 at 6:59
  • $\begingroup$ Just a small doubt about the continuous part, when we change the cost function a little how does the neuron output change. We will be measuring the output of the neuron value directly instead of whether it fired or not? Am I right? $\endgroup$
    – Kira
    May 16, 2017 at 15:38
  • 1
    $\begingroup$ You do not change the cost function; instead you change the weights and biases which leads to a change in the cost. And in practice you use sigmoid neurons, again for smoothness reasons to avoid flipping, as described in your linked article $\endgroup$
    – Henry
    May 16, 2017 at 16:07

1 Answer 1

Reset to default
3
$\begingroup$

We have $$ \frac{d}{dx}x^2 = 2x $$ which is obviously smooth.

But the accuracy function is $f(\hat{y})=\mathbb{I}(y = \text{max}_i \hat{y}_i)$, using the convention that $y$ is an integer label and $\hat{y}$ is vector predicting the label. Clearly $f$ is a step function: constant right up until a different value is the maximum. Step functions are not smooth.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.