8
$\begingroup$

I am trying to get my head around word2vec (paper) and the underlying Skip-gram model. I hope I got the basics and intuition, but I am still not sure whether bias units are used in the input and/or in the hidden layer.

The input is just a one-hot encoded vector and it is often said it just serves as a selector for the weights associated with the corresponding word (there is no activation function). I would say, there is no bias unit added to the input layer. Now as for the hidden layer, since the output neurons give the following:

softmax activation

where v' and v are "input and output representation of w" I don't think there is a bias unit either.

In case I am right, why is there no need for bias units in this type of neural network? In case I am wrong, can anyone explain how do they fit into the description of the model?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
+50
$\begingroup$

It seems there are indeed no bias units at either layer. In his thesis on neural network based language models, Mikolov states that:

[...] biases are not used in the neural network, as no significant improvement of performance was observed - following the Occam's razor, the solution is as simple as it needs to be.

(Mikolov, T.: Statistical Language Models Based on Neural Networks, p. 29)

While this is a quote concerning recurrent neural networks specifically, I am going to assume the same is valid for the Skip-gram model.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you for linking this. But after digging into the underlying math, I think this depends on the specific inferences that you want to make. If you only care about calculating word similarity, this is totally justifiable since the bias drop out due to translation invariance of the distance metric. But if you are interested in calculating something else, you still need that bias. $\endgroup$
    – mortonjt
    Aug 23, 2018 at 20:18
0
$\begingroup$

Bias is hidden in average vectors (take mean of all vectors; projection of a given vector on this mean effectively carries a bias).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.