Is skip-gram model of word embedding actually a multi-class task not a multi-label task, right?

Question

So curious about this question, that I can't describe it in short. Please forgive me.

Description：

From multiclass and multilabel algorithms, we can get the definition of the multi-class and multi-label task.

Multiclass classification means a classification task with more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Multiclass classification makes the assumption that each sample is assigned to one and only one label: a fruit can be either an apple or a pear but not both at the same time.

Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A text might be about any of religion, politics, finance or education at the same time or none of these.

Intuitively, a word can appear with multi-words simultaneously, so the skip-gram model should be regarded as a multi-label task. But actually we treat it as a multi-class task according to the implementation of tensorflow.

I've found some difference between these two tasks:

1. The activation function of last layer is different. For multi-class task, is softmax, which means the probability of one word relates to other words, and sum of the probabilities is 1, from which we can get the degree of preference of that word. But for multi-label task, is sigmoid, which means whether the word relates to other words, the output also probabilities, but they are all between 0 to 1, from which we can't get the degree of preference of that word.
2. The training data's label is different. For multi-class task, is one-hot, which means one word pair is a training data, and one target word can generate multiple same training data when it appears with other words many times. But for multi-label task, is multi-hot, which means we should gather all the word's related word to generate label, and one target word can produce just one training data no matter how many times it appears with other words.
3. From the description above， we can see that the training effectiveness of two task must be different. In my opinion Using more training data, multi-class task should get the more accurate result.

Questions:

1. Is skip-gram model actually a multi-class task?
2. If question 1 is yes, why don't we regard it a multi-label task? What are the disadvantages of the multi-label task?
3. Do the negative sample and nce loss of skip-gram model just design for the computation consideration?

Looking forward to your insights, thanks very much~

Answer 1

1. Is skip-gram model actually a multi-class task?

I don't think it is a multi-class task, and you can see that it does not use the softmax but the NCE because the normalization of the softmax is too expensive.

2. If question 1 is yes, why don't we regard it a multi-label task? What are the disadvantages of the multi-label task?

NCE is in nature a multilabel classification(but with only 1 positive label and k negative ones), and you can refer to the definition from the original paper function 11.

3. Do the negative sample and nce loss of skip-gram model just design for the computation consideration?

Yes. For the negative labels we don't use all other words in the vocabulary but sample k words randomly.