We know that usually the output of neutral network is a value. If the output is a vector such as the multi-steps prediction in time series, what's the loss function of vector? Is it simply the sum of the loss functions of each component?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ The loss function is whatever you define it to be. It doesn't necessarily need to break down component-wise. $\endgroup$– Arya McCarthyFeb 24, 2021 at 17:49
-
$\begingroup$ Depends on the loss function. Some losses are defined in terms of vectors, and then give a scalar for each vector. Some will just compute a scalar loss for each element and then sum/average the elements to give a single loss value. $\endgroup$– Sycorax ♦Feb 24, 2021 at 18:01
-
$\begingroup$ @Sycorax could you give the examples for each case as an answer? Since I never contact with the loss of vector output before. $\endgroup$– user6703592Feb 24, 2021 at 18:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Depends on the loss function. Some losses are defined in terms of vectors, and then give a scalar for each vector. Some will just compute a scalar loss for each element and then sum/average the elements to give a single loss value.
Triplet losses are used to train embedding vectors $f(x_i)$, so for each input (an image, a text, something else), the network returns a vector. The loss $L$ for a sample $x^a$ is computed in terms of distances to the embeddings of two other samples $x^p$ and $x^n$ .
$$\begin{aligned} L &= \max\left\{0, \left\|f(x^a) - f(x^p) \right\|_2^2 - \left\|f(x^a)-f(x^n)\right\|_2^2 +\alpha\right\} \end{aligned} $$
Other losses just suppress the distinction between vector and scalar values. Suppose that your network takes an $n \times m$ pixel image as an input and gives an $n \times m$ matrix output as a prediction (for instance, if you're computing a boolean mask of the input image). For some task, it might be sufficient to flatten the output and the mask (label array) and compute the softmax loss in the usual way.
