8
$\begingroup$

Problem Description

I'm beginning network construction for a problem that I feel could have a far more insightful loss function than a simple MSE regression.

My problem deals with multi-category classification (see my question on SO for what I mean by this), where there is a defined distance or relation between categories that should be taken into account.

Another point is that the error should not be effected by the number of firing categories present. I.e. the error for 5 firing categories each off by 0.1, should be the same as 1 firing category off by 0.1. (by firing I mean that they are non-zero, or above some threshold)

Key points

  • multi-category classification (multiple firing at once)
  • relationships between categories
  • count of firing categories should not effect loss:
    • enter image description here

My Attempt

Mean Squared Error seems like a good place to start:

enter image description here

This is simply considering category-by-category, which is still valuable in my problem but misses a large portion of the picture.

enter image description here

Here is my attempt to rectify the idea of distance between categories. Next I would like to take into account the number of categories firing (call it: v)

enter image description here

My Question

I have a very weak background in statistics; as a result, I don't have many tools in my belt to broach a problem like this. The umbrella topic of what I'm asking would appear to be "When forming a cost function, how does one go about combining multiple measure of cost? Or what techniques can one apply to do so?". I would also appreciate having any flaws in my thought process exposed and improved upon.

I value being taught why my mistakes are mistakes, as opposed to having someone solely correct them without explanation.

If any piece of this question lacks clarity or could be improved, please let me know.

Improve this question
$\endgroup$
1
  • $\begingroup$ Aidan, it's good to see so much thought devoted to building a problem-specific loss function. I would be inclined to view this as a math problem, rather than a statistics problem. You are searching for a loss function that takes 2x5 matrices to the real numbers, and you have some strong ideas about certain invariants this function ought to satisfy, which therefore impose constraints on the functional form. If you would explain the meaning of your matrices, I could probably offer some more specific guidance for building your loss function. $\endgroup$
    – David C. Norris
    Commented Dec 2, 2015 at 17:30

1 Answer 1

Reset to default
2
+50
$\begingroup$

You can use hinge loss which is an upper bound on the classification loss; that is, it penalizes the model if the label of the highest scoring category is different from the label of the ground-truth class.

For more details on the relation between classification loss and hinge loss you can read Section 2 of this awesome paper from C.N. J. Yu and T. Joachims.

In summary, there is a task loss, usually denoted by $\Delta \left( y_i, \hat{y}(x_i) \right)$, which measures the penalty for predicting output $\hat{y}(x_i)$ for input $x_i$ when the expected (ground-truth) output is $y_i$. The task loss for multi-class classification is usually defined as $\Delta \left( y_i, \hat{y}(x_i) \right) = \mathbf{1}\{ y_i \neq \hat{y}(x_i) \}$. However, as long as $\Delta$ only depends on the two labels $y$ and $\hat{y}$, you can define it however you want. In particular, one can view $\Delta$ as an arbitrary $K \times K$ matrix where $K$ is the number of categories and $\Delta(a, b)$ indicates the penalty of classifying an input of category $a$ as belonging to category $b$.

For example: $\\\text{input data}: \\ \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}, \quad x_i \in \mathbb{R}^d, \quad y_i \in \mathcal{Y}=\{c_1, c_2, c_3, c_4\} \\ \text{network predictions}:\\ \hat{y}(x_1)=c_2, \quad \hat{y}(x_2)=c_1, \quad \hat{y}(x_3)=c_3 \\ \text{task loss matrix}:\\ \begin{bmatrix} \Delta(y_{1}, y_{1}) & \Delta(y_{1}, y_{2}) & \Delta(y_{1}, y_{3}) & \Delta(y_{1}, y_{4}) \\ \Delta(y_{2}, y_{1}) & \Delta(y_{2}, y_{2}) & \Delta(y_{2}, y_{3}) & \Delta(y_{2}, y_{4}) \\ \Delta(y_{3}, y_{1}) & \Delta(y_{3}, y_{2}) & \Delta(y_{3}, y_{3}) & \Delta(y_{3}, y_{4}) \\ \Delta(y_{4}, y_{1}) & \Delta(y_{4}, y_{2}) & \Delta(y_{4}, y_{3}) & \Delta(y_{4}, y_{4}) \end{bmatrix} = \begin{bmatrix} 0 & 1 & 2 & 3 \\ 1 & 0 & 1 & 2 \\ 2 & 1 & 0 & 1 \\ 3 & 2 & 1 & 0 \end{bmatrix} \\ \text{classification loss assuming $\quad y_1=c_4, \quad y_2=c_1, \quad y_3=c_4$:} \\ \Delta(y_1, \hat{y}(x_1)) = \Delta(c_4, c_2) = 2 \\ \Delta(y_2, \hat{y}(x_2)) = \Delta(c_1, c_1) = 0 \\ \Delta(y_3, \hat{y}(x_3)) = \Delta(c_4, c_3) = 1 \\ $

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ thanks so much for the response. I've added an example to you question (may still be in peer-review when you see this comment). Can you confirm that my interpretation is correct? $\endgroup$
    – Aidan Gomez
    Commented Dec 3, 2015 at 23:10
  • $\begingroup$ I see, this is valuable in a classification-style problem, but mine is regression (with multi-dimensional labels), where multiple categories may be "on" at the same time. This seems to be similar to an argmax; for each input it only considers the largest output category. In my problem one might have a label like {1,0,1,1} where categories 0, 2, 3 are all present in the data but category 1 is not. If my network guessed {0.8, 0, 0.6, 0.3}, my loss shouldn't be the same as a guess like {0.8, 0, 0.7, 0.7}. $\endgroup$
    – Aidan Gomez
    Commented Dec 4, 2015 at 0:03
  • $\begingroup$ I still believe my response answers your question! What I explained is called Structural SVM where the label space $\mathcal{Y}$ (i.e. the space that the ground-truth labels live in) can have any structure. It seems to me that what you want is can be obtained as follows: consider a classification problem with $K$ categories; define $\mathcal{Y} = \{0, 1\}^K$. The only thing is that the size of your $\Delta$ matrix then becomes $2^K \times 2^K$. However, you may not need to specify the matrix. Feel free to ask for more detail if this sounds to answer your question. $\endgroup$
    – Sobi
    Commented Dec 4, 2015 at 0:39
  • $\begingroup$ This sounds like the correct path, could you provide an easy example like we did for the 1-dimensional label case? Perhaps for K=2 or 3 $\endgroup$
    – Aidan Gomez
    Commented Dec 4, 2015 at 0:41
  • $\begingroup$ Let's say for an input $x$ the correct answer is $y=(0, 1, 1)$ but your classifiers predicts $\hat{y}(x)=(0.1, 0.9, 0.8)$. In this case I am assuming $\Delta: \{0, 1\}^3 \times [0, 1]^3 \rightarrow \mathbb{R}$. You can define the loss function to be , for example, $\Delta(y, \hat{y}) = \max_{k=1}^K |y[k] - \hat{y}[k]|$; this looks at the predictions of the classifier for all classes and returns the loss value of the class which is the farthest from it's corresponding ground-truth value. $\endgroup$
    – Sobi
    Commented Dec 4, 2015 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.