0
$\begingroup$

Suppose one sample of my training data consists of a sequence with $n$ elements. My task is to do binary classification on one element in the sequence, and my labels are such that for each sequence in the training set, I only have a label for one element which I'll call the 'target' element. I use self-attention layers to create embeddings for each of the $n$ elements in one sequence. I then want to run the embedding, for the target element only, through a feed-forward network resulting in a binary prediction. The index for the target element can be different for each sample. The target element is determined in advance (it's not a function of the neural network parameters) I want to know if it makes sense to backpropagate on a loss which is calculated from the embedding for one element only? And to get some intuition or theoretical justification for why this approach is valid or not.

To give an analogy: my sequences are sentences and my task is to make a binary prediction on one word in each sentence and I have a dataset that determines the target word for each sentence and its corresponding label. The goal of the model is not to predict which word is the target, but rather to assign the correct label to a pre-specified target.

The motivation is that I have a pre-trained model which already creates good embeddings for each element, I want to fine-tune the per-element embeddings on a much smaller task-specific dataset and I suspect that trying to aggregate all the embeddings into a single embedding that summarizes the whole sequence will dilute the signal from the target element.

$\endgroup$

1 Answer 1

0
$\begingroup$

Having thought about this for a few days, it's clear this is a valid loss function for gradient descent provided that the loss function is differentiable with respect to the parameters of the model.

The 'selection' operation that is used to calculate the loss on the target element can be achieved by taking a one-hot-encoded vector (dimensions $(1, n)$ where $n$ is the number of elements) where the $1$ indicates the position of the target element. And multiplying the sequence representation (dimensions $(n,k)$ where $k$ is the number of features) by this one-hot-encoded vector. Given that this selection operation is simply a matrix multiplication and the one-hot selection vector does not depend on the model parameters it is clearly differentiable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.