Word2vec only have one hidden layer followed by a softmax layer. If we add more hidden layer(fully connected feed forward layers), then the model complexity is increased and likely we will get a more powerful model. As far as I know, DAN(deep averaging network) uses 2 feed forward layers. But why we haven't seen such models in word2vec published?
1
-
2$\begingroup$ In the context of its time (Mikolov's paper was published in 2013), the big breakthrough in word2vec is that it was cheap & fast, which meant that it was viable to apply it to massive datasets. Computing hardware and NLP model architectures have both improved since then, making alternative models viable. As of 2022, the clear favorites for achieving state-of-the-art results in NLP tasks are transformer networks. But for any particular application, the question should be "Does this model solve my problem?" which can involve efficiency, cost, and accuracy considerations. $\endgroup$– Sycorax ♦Mar 5 at 19:34
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Word2vec is popular because it's simple while being good enough. Because of being simple, it's fast and needs less expensive hardware to run. Yes, using one vs two layers does not seem to make much difference, but if you need to run it in production environment, the all those milliseconds can add to something quite big over millions of calls. There are a lot of NLP deep neural networks, including huge ones. It's about picking the right tool for the job, you don't need a laser knife for a job doable by a rusted axe.
$\begingroup$
$\endgroup$
1
Probably because the model becomes more complicated than necessary. While deep networks can accommodate more non-linearity which makes them more powerful in certain tasks, adding more non-linearity than necessary will increase training cost with no significant improvement in performance. Deep Averaging Networks (DANs) are used to obtain sentence/document embeddings, which is a much more complex task than obtaining word embeddings, so adding more hidden layers is required. Recent models focus on using the input in more intelligent ways, such as using attention. They usually use 1-2 hidden layers.
Also see this question can we make a word2vec nn of more than 3 layers using tensorflow.
-
$\begingroup$ Thanks for sharing. I would assume increasing for example, two hidden layers won’t leads too much computational power(double amount of parameters). I don’t even see many literature talking about the performance of increasing the model complexity though. My prediction will be the model performance will be better, given there are so many datasets and computational cost is acceptable. It is possible that extra layers is not necessary, but at least doing some experiments will be helpful while I don’t see any documents. If you have any references, please kindly let me know. $\endgroup$ Feb 12, 2020 at 18:34