Quadratic error for multi-class classification

Question

I'm trying to train a neural network to classify handwritten inputs into 10 categories, each for one digit (1,...,9,0). I represent the output of an example using a 10-dimensional vector. Digit 5, for example, is represented like:

$$\begin{bmatrix} 0.0\\ 0.0\\ 0.0\\ 0.0\\ 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{bmatrix}$$

However, I'm not sure how should I compute the quadratic error of the output of my network. When working with single-class classification, I simply compute:

$$quaderror(o, t) = \frac{1}{2}(o - t)^2$$

Where $o$ is the output of my network and $t$ is the target output. What should the quadratic error be in the case of multi-class classification? Should it be the quadratic error for each term? Or the square of the norm of the difference? Or something else entirely? What is the derivative of the error with respect to $o$ in this case?

Answer 1

Error just describe how wrong is your solution. It would be zero value if it's right and 2 otherwise. The important thing that you will minimize value if it was wrong, but if your difference would be zero than you didn't get any update for right answer. Last one looks OK, but if you think about it, there are can be situation when your winner neuron can be extremely close to some other two and small updates in input data can change your network answer. For a good learning process for classification tasks the best way is to learn probabilities. For this issue you must put Softmax layer as final one and instead of competitive. In this case your error will show not only how your network guess the result, but also how confidence it in this result.

Also you can use cross entropy error function, but for it you must use Softmax layer as final one.