I'm trying to train a neural network to classify handwritten inputs into 10 categories, each for one digit (1,...,9,0). I represent the output of an example using a 10-dimensional vector. Digit 5, for example, is represented like:
$$\begin{bmatrix} 0.0\\ 0.0\\ 0.0\\ 0.0\\ 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{bmatrix}$$
However, I'm not sure how should I compute the quadratic error of the output of my network. When working with single-class classification, I simply compute:
$$quaderror(o, t) = \frac{1}{2}(o - t)^2$$
Where $o$ is the output of my network and $t$ is the target output. What should the quadratic error be in the case of multi-class classification? Should it be the quadratic error for each term? Or the square of the norm of the difference? Or something else entirely? What is the derivative of the error with respect to $o$ in this case?