Constructing a problem-specific loss function Problem Description
I'm beginning network construction for a problem that I feel could have a far more insightful loss function than a simple MSE regression.
My problem deals with multi-category classification (see my question on SO for what I mean by this), where there is a defined distance or relation between categories that should be taken into account.
Another point is that the error should not be effected by the number of firing categories present. I.e. the error for 5 firing categories each off by 0.1, should be the same as 1 firing category off by 0.1. (by firing I mean that they are non-zero, or above some threshold)

Key points

*

*multi-category classification (multiple firing at once)

*relationships between categories

*count of firing categories should not effect loss:

*

*

My Attempt
Mean Squared Error seems like a good place to start:

This is simply considering category-by-category, which is still valuable in my problem but misses a large portion of the picture.

Here is my attempt to rectify the idea of distance between categories. Next I would like to take into account the number of categories firing (call it: v)


My Question
I have a very weak background in statistics; as a result, I don't have many tools in my belt to broach a problem like this. The umbrella topic of what I'm asking would appear to be "When forming a cost function, how does one go about combining multiple measure of cost? Or what techniques can one apply to do so?". I would also appreciate having any flaws in my thought process exposed and improved upon.
I value being taught why my mistakes are mistakes, as opposed to having someone solely correct them without explanation.
If any piece of this question lacks clarity or could be improved, please let me know.
 A: You can use hinge loss which is an upper bound on the classification loss; that is, it penalizes the model if the label of the highest scoring category is different from the label of the ground-truth class.
For more details on the relation between classification loss and hinge loss you can read Section 2 of this awesome paper from C.N. J. Yu and T. Joachims. 
In summary, there is a task loss, usually denoted by $\Delta \left( y_i, \hat{y}(x_i) \right)$, which measures the penalty for predicting output $\hat{y}(x_i)$ for input $x_i$ when the expected (ground-truth) output is $y_i$. The task loss for multi-class classification is usually defined as $\Delta \left( y_i, \hat{y}(x_i) \right) = \mathbf{1}\{ y_i \neq \hat{y}(x_i) \}$. However, as long as $\Delta$ only depends on the two labels $y$ and $\hat{y}$, you can define it however you want. In particular, one can view $\Delta$ as an arbitrary $K \times K$ matrix where $K$ is the number of categories and $\Delta(a, b)$ indicates the penalty of classifying an input of category $a$ as belonging to category $b$.
For example:
$\\\text{input data}: \\
\{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}, \quad x_i \in \mathbb{R}^d, \quad y_i \in \mathcal{Y}=\{c_1, c_2, c_3, c_4\} \\
\text{network predictions}:\\ 
\hat{y}(x_1)=c_2, \quad \hat{y}(x_2)=c_1, \quad \hat{y}(x_3)=c_3 \\ 
\text{task loss matrix}:\\ 
\begin{bmatrix} 
\Delta(y_{1}, y_{1}) & \Delta(y_{1}, y_{2}) & \Delta(y_{1}, y_{3}) & \Delta(y_{1}, y_{4}) \\ 
\Delta(y_{2}, y_{1}) & \Delta(y_{2}, y_{2}) & \Delta(y_{2}, y_{3}) & \Delta(y_{2}, y_{4}) \\ 
\Delta(y_{3}, y_{1}) & \Delta(y_{3}, y_{2}) & \Delta(y_{3}, y_{3}) & \Delta(y_{3}, y_{4}) \\ 
\Delta(y_{4}, y_{1}) & \Delta(y_{4}, y_{2}) & \Delta(y_{4}, y_{3}) & \Delta(y_{4}, y_{4}) 
\end{bmatrix}
=
\begin{bmatrix} 
0 & 1 & 2 & 3 \\ 
1 & 0 & 1 & 2 \\ 
2 & 1 & 0 & 1 \\ 
3 & 2 & 1 & 0 
\end{bmatrix} 
\\
\text{classification loss assuming $\quad y_1=c_4, \quad y_2=c_1, \quad y_3=c_4$:} \\
\Delta(y_1, \hat{y}(x_1)) = \Delta(c_4, c_2) = 2 \\ 
\Delta(y_2, \hat{y}(x_2)) = \Delta(c_1, c_1) = 0 \\ 
\Delta(y_3, \hat{y}(x_3)) = \Delta(c_4, c_3) = 1 \\
$
