3
$\begingroup$

We know that usually the output of neutral network is a value. If the output is a vector such as the multi-steps prediction in time series, what's the loss function of vector? Is it simply the sum of the loss functions of each component?

$\endgroup$
3
  • $\begingroup$ The loss function is whatever you define it to be. It doesn't necessarily need to break down component-wise. $\endgroup$ Feb 24, 2021 at 17:49
  • $\begingroup$ Depends on the loss function. Some losses are defined in terms of vectors, and then give a scalar for each vector. Some will just compute a scalar loss for each element and then sum/average the elements to give a single loss value. $\endgroup$
    – Sycorax
    Feb 24, 2021 at 18:01
  • $\begingroup$ @Sycorax could you give the examples for each case as an answer? Since I never contact with the loss of vector output before. $\endgroup$ Feb 24, 2021 at 18:33

1 Answer 1

2
$\begingroup$

Depends on the loss function. Some losses are defined in terms of vectors, and then give a scalar for each vector. Some will just compute a scalar loss for each element and then sum/average the elements to give a single loss value.

Triplet losses are used to train embedding vectors $f(x_i)$, so for each input (an image, a text, something else), the network returns a vector. The loss $L$ for a sample $x^a$ is computed in terms of distances to the embeddings of two other samples $x^p$ and $x^n$ .

$$\begin{aligned} L &= \max\left\{0, \left\|f(x^a) - f(x^p) \right\|_2^2 - \left\|f(x^a)-f(x^n)\right\|_2^2 +\alpha\right\} \end{aligned} $$

See also In training, I first have a solid drop in loss, but eventually the loss slowly but consistently increases. What could cause this?

Other losses just suppress the distinction between vector and scalar values. Suppose that your network takes an $n \times m$ pixel image as an input and gives an $n \times m$ matrix output as a prediction (for instance, if you're computing a boolean mask of the input image). For some task, it might be sufficient to flatten the output and the mask (label array) and compute the softmax loss in the usual way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.