I'm looking to use a feedforward neural network for classification where some mistakes are worse than others and some correct predictions are better than others. In particular, for each sample and its possible classification, I'd like to incorporate a signed weight roughly corresponding to how good or bad a prediction is.

I originally naively attempted this by using theano's "categorical crossentropy" cost function (here) with p(x)/true_dist acting as the weights, but it didn't work well. I now realize is bc my weights are not strictly positive and do not sum to 1. I've also considered a weighted squared error cost function, but that seems likely to be very slow to train without a log in the cost function. A literature search has not turned up anything, but perhaps I just don't know the proper terms to look for.

Can anyone point me in the right direction? What is the standard way to tackle this problem or if it hasn't been done before can you recommend a good cost function or network setup to accomplish my desired weighting scheme?

  • $\begingroup$ Why is training a weighted squared-error cost function much slower to train than a unqeighted square-error cost function? $\endgroup$ – dimpol Nov 14 '16 at 12:37
  • $\begingroup$ The derivative of the squared-error cost function with respect to the weights depends on the derivative of the activation function, which if you use sigmoid or softmax activation functions, can be very small for particularly large or small values and thus slow to train. Meanwhile, the gradient of the crossentropy cost function with respect to the weights does not include this term and instead depends on the difference between the predicted value and the target value, which trains more quickly. For a more thorough discussion of this see neuralnetworksanddeeplearning.com/chap3.html $\endgroup$ – ChadZ11 Nov 14 '16 at 20:58
  • $\begingroup$ But why do you want to train the weights of the squared error-term? Aren't those weights based on the problem statement? How and why do you want to train how bad one mistake is over another mistake? $\endgroup$ – dimpol Nov 15 '16 at 8:55
  • $\begingroup$ I think we've miscommunicated. Correct, I don't want to train the weights of the squared error cost function as I have them beforehand, and they are specific to my problem. I was referring to the derivative of the cost function with respect to the weights in my output layer that most directly affect the network's output which is directly used to calculate the cost function. I think the link in my comment above does a good job of explaining. $\endgroup$ – ChadZ11 Nov 16 '16 at 5:17
  • 1
    $\begingroup$ Ah, now I understand. Can't you just add those weights to the elements of the sum of the cross-entropy function? $\endgroup$ – dimpol Nov 16 '16 at 10:44

The comment thread hints at the correct answer -- since your different samples have different levels of "importance" to your task, you should assign larger weights to the higher-importance samples.

For a usual cross-entropy loss, we have labels $y$ and predictions $\hat{y}\in(0,1)$. So the loss for a single sample is $$ -\left[y \log(\hat{y}) + (1 - y)\log(1 -\hat{y})\right] $$ but weighted cross-entropy has positive weights $w$. The loss for a single sample is: $$ -w \left[y \log(\hat{y}) + (1 - y)\log(1 -\hat{y})\right] $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.