I'm looking to use a feedforward neural network for classification where some mistakes are worse than others and some correct predictions are better than others. In particular, for each sample and its possible classification, I'd like to incorporate a signed weight roughly corresponding to how good or bad a prediction is.
I originally naively attempted this by using theano's "categorical crossentropy" cost function (here) with p(x)/true_dist acting as the weights, but it didn't work well. I now realize is bc my weights are not strictly positive and do not sum to 1. I've also considered a weighted squared error cost function, but that seems likely to be very slow to train without a log in the cost function. A literature search has not turned up anything, but perhaps I just don't know the proper terms to look for.
Can anyone point me in the right direction? What is the standard way to tackle this problem or if it hasn't been done before can you recommend a good cost function or network setup to accomplish my desired weighting scheme?