0
$\begingroup$

While revising my notes that i took in a deep learning related course (in calculating gradients step), I see "we assume our loss function is a decoupled function". Can someone explain what this sentence means? I do not remember at all.

Edit 1 - more explanation: I also noted that in order to be able to apply backpropagation, our loss function should be decoupled in the sense that only one value (average loss) is used to backpropagate the error. We do not use a vector of loss that contains the loss of each data point in a mini-batch for example. Instead, I noted that "we need a decoupled loss function to be able to apply backpropagation". Why?

Improve this question
$\endgroup$
4
  • $\begingroup$ Where is this from? $\endgroup$
    – Aksakal
    Dec 21, 2021 at 18:14
  • 1
    $\begingroup$ More context is necessary $\endgroup$
    – Aksakal
    Dec 21, 2021 at 19:22
  • $\begingroup$ I tried to add more explaination. $\endgroup$
    – Mas A
    Dec 22, 2021 at 13:35
  • $\begingroup$ Doing an internet search on decoupled and backprop leads to papers describing how to implement a form of backprop which can be performed on multiple nodes simultaeously (perhaps except for the last) which allows parallel gradient computation which greatly speeds up computation for deep learning with many tens of layers in the network. Perhaps that is what you intended? $\endgroup$
    – Avraham
    Dec 22, 2021 at 14:08

1 Answer 1

Reset to default
4
$\begingroup$

I think "decoupled function" is meant in the sense of this preprint: https://arxiv.org/pdf/1805.08479.pdf where they discuss multivariate functions being "decoupled" into the sum of univariate functions.

If you have $N$ data points, your loss function should be written as the sum (or average) of $N$ univariate functions, which are functions only of the predicted value at that point. For example, sum of squared errors is decoupled, since the total loss is just the sum of the loss at each point.

This would be violated if you have some loss function which considers explicit dependence between the observations. The condition is sort of like assuming your data are IID - it's the IID assumption which allows us to work with decoupled loss functions.

For example, if you tried to build a model with this loss function: $$\sum_i\sum_j (\hat{y_i} - \hat{y_j})^2$$, which minimizes the difference between all predictions, and says nothing else about what they should be. In order to calculate the gradient of $\hat{y_i}$ as a function of the parameters in the model, you'd also need to know the value of every $\hat{y_j}$. So backprop isn't possible.

Improve this answer
$\endgroup$
6
  • $\begingroup$ With this iid assumption part, it makes more sense to me. Therefore the important part is not about having something at the end (loss as a scalar), instead, the point is assuming the independence between the results of the loss of each data point. In that sense, can we say that softmax is not a decoupled function, don't we? $\endgroup$
    – Mas A
    Dec 22, 2021 at 15:16
  • $\begingroup$ Well, softmax is an activation function, not a loss function. Cross-entropy is usually used as a loss function when softmax is the final output layer, and you do have a scalar loss value for each observation from cross-entropy. $\endgroup$
    – Jonny Lomond
    Dec 22, 2021 at 16:48
  • $\begingroup$ @JonnyLomond This is a good answer. Would triplet-loss be an example of a loss function that is not decoupled? stats.stackexchange.com/questions/475655/… $\endgroup$
    – Sycorax ♦
    Dec 22, 2021 at 17:17
  • $\begingroup$ @Sycorax I think it would not be decoupled if each "observation" were a single image and the loss were written in terms of the difference in predicted value between any 3 observations, but they take observations to be triplets of images. The optimizer behaves like it is searching through functions $h(A,B,C)$. It happens that $h$ can always be written $h(A,B,C) = (f(A), f(B), f(C))$ and we actually care about $f$, but that's imposed by the network structure and the optimizer doesn't know about it, so it's decoupled. $\endgroup$
    – Jonny Lomond
    Dec 22, 2021 at 18:36
  • $\begingroup$ In this notation, is $f(X)$ the vector assigned to the input and $h$ is the loss function? Or something else? $\endgroup$
    – Sycorax ♦
    Dec 22, 2021 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.