0
$\begingroup$

Word2vec only have one hidden layer followed by a softmax layer. If we add more hidden layer(fully connected feed forward layers), then the model complexity is increased and likely we will get a more powerful model. As far as I know, DAN(deep averaging network) uses 2 feed forward layers. But why we haven't seen such models in word2vec published?

$\endgroup$
0
$\begingroup$

Probably because the model becomes more complicated than necessary. While deep networks can accommodate more non-linearity which makes them more powerful in certain tasks, adding more non-linearity than necessary will increase training cost with no significant improvement in performance. Deep Averaging Networks (DANs) are used to obtain sentence/document embeddings, which is a much more complex task than obtaining word embeddings, so adding more hidden layers is required. Recent models focus on using the input in more intelligent ways, such as using attention. They usually use 1-2 hidden layers.

Also see this question can we make a word2vec nn of more than 3 layers using tensorflow.

$\endgroup$
1
  • $\begingroup$ Thanks for sharing. I would assume increasing for example, two hidden layers won’t leads too much computational power(double amount of parameters). I don’t even see many literature talking about the performance of increasing the model complexity though. My prediction will be the model performance will be better, given there are so many datasets and computational cost is acceptable. It is possible that extra layers is not necessary, but at least doing some experiments will be helpful while I don’t see any documents. If you have any references, please kindly let me know. $\endgroup$ – essence16 Feb 12 '20 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.