90

In fact, the output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all posible vectors. That is the reason why you need to specify the size of the vocabulary as the first argument (so the table can be initialized). The most common application of this ...


69

Relation to Word2Vec ========================================== Word2Vec in a simple picture: More in-depth explanation: I believe it's related to the recent Word2Vec innovation in natural language processing. Roughly, Word2Vec means our vocabulary is discrete and we will learn an map which will embed each word into a continuous vector space. Using this ...


30

When the downstream applications only care about the direction of the word vectors (e.g. they only pay attention to the cosine similarity of two words), then normalize, and forget about length. However, if the downstream applications are able to (or need to) consider more sensible aspects, such as word significance, or consistency in word usage (see below), ...


27

The issue There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors are learned while the network learns to predict the next word given a window of words (the input of the network). Predicting the next word is like predicting the class. That is, such a network is just a "standard" multinomial (...


23

One simple technique that seems to work reasonably well for short texts (e.g., a sentence or a tweet) is to compute the vector for each word in the document, and then aggregate them using the coordinate-wise mean, min, or max. Based on results in one recent paper, it seems that using the min and the max works reasonably well. It's not optimal, but it's ...


18

Embedding layers in Keras are trained just like any other layer in your network architecture: they are tuned to minimize the loss function by using the selected optimization method. The major difference with other layers, is that their output is not a mathematical function of the input. Instead the input to the layer is used to index a table with the ...


16

You can use doc2vec similar to word2vec and use a pre-trained model from a large corpus. Then use something like .infer_vector() in gensim to construct a document vector. The doc2vec training doesn't necessary need to come from the training set. Another method is to use an RNN, CNN or feed forward network to classify. This effectively combines the word ...


14

In CBOW the vectors from the context words are averaged before predicting the center word. In skip-gram there is no averaging of embedding vectors. It seems like the model can learn better representations for the rare words when their vectors are not averaged with the other context words in the process of making the predictions.


13

Footnote at http://arxiv.org/abs/1412.5335 (one of the authors is Tomas Mikolov) says In our experiments, to match the results from (Le & Mikolov, 2014), we followed the suggestion by Quoc Le to use hierarchical softmax instead of negative sampling. However, this produces the 92.6% accuracy result only when the training and test data are not shuffled. ...


13

Here is my oversimplified and rather naive understanding of the difference: As we know, CBOW is learning to predict the word by the context. Or maximize the probability of the target word by looking at the context. And this happens to be a problem for rare words. For example, given the context yesterday was really [...] day CBOW model will tell you that ...


12

I highly reccomend the article How to Use t-SNE Effectively. It has great animated plots of the tsne fitting process, and was the first source that actually gave me an intuitive understanding of what tsne does. At a high level, perplexity is the parameter that matters. It's a good idea to try perplexity of 5, 30, and 50, and look at the results. But ...


11

according to Dan Jurafsky and James H. Martin book: "It turns out, however, that simple frequency isn’t the best measure of association between words. One problem is that raw frequency is very skewed and not very discriminative. If we want to know what kinds of contexts are shared by apricot and pineapple but not by digital and information, we’re not going ...


10

I'm not an expert in word2vec, but upon reading Rong, X. (2014). word2vec Parameter Learning Explained and from my own NN experience, I'd simplify the reasoning to this: Hierarchical softmax provides for an improvement in training efficiency since the output vector is determined by a tree-like traversal of the network layers; a given training sample only ...


9

Option 1 (adding an unknown word token) is how most people solve this problem. Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained. Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional ...


7

I had the same problem understanding it. It seems that the output score vector will be the same for all C terms. But the difference in error with each one-hot represented vectors will be different. Thus the error vectors are used in back-propagation to update the weights. Please correct me, if I'm wrong. source : https://iksinc.wordpress.com/tag/skip-gram-...


7

Garten et al. {1} compared word vectors obtained by adding input word vectors with output word vectors, vs. word vectors obtained by concatenating input word vectors with output word vectors. In their experiments, concatenating yield significantly better results: The video lecture {2} recommends to average input word vectors with output word vectors, but ...


6

If you are working with English text and want pre-trained word embeddings to begin with, then please see this: https://code.google.com/archive/p/word2vec/ This is the original C version of word2vec. Along with this release, they also released a model trained on 100 billion words taken from Google News articles (see subsection titled: "Pre-trained word and ...


6

Figure 1 there clarifies things a bit. All word vectors from window of a given size are summed up, result is multiplied by (1/window size) and then fed into output layer. Projection matrix means a whole lookup table where each word corresponds to single real-valued vector. Projection layer is effectivly a process that takes a word (word index) and returns ...


6

The contextual embedding of a word is just the corresponding hidden state of a bi-GRU: In our model the document encoder $f$ is implemented as a bidirectional Gated Recurrent Unit (GRU) network whose hidden states form the contextual word embeddings, that is $f_i(d) = \overrightarrow{f_i}(d) \,\, ||\,\, \overleftarrow{f_i}(d)$, where $||$ denotes vector ...


6

what are my actual word vectors in the end? The actual word vectors are the hidden representations $h$ Basically, multiplying a one hot vector with $\mathbf{W_{V\times N}}$ will give you a $1$$\times$$N$ vector which represents the word vector for the one hot you entered. Here we multiply the one hot $1$$\times$$5$ for say 'chicken' with synapse 1 $\mathbf{...


6

I also had the same question and after reading a couple of posts and materials I think I figured out what embedding layer role is. I think this post is also helpful to understand, however, I really find Daniel's answer convenient to digest. But I also got the idea behind it mainly by understanding the embedding words. I believe it's inaccurate to say ...


6

This quote is clearly talking about sentence embeddings, obtained from word embeddings. If the sentence $s$ consists of words $(w_1, ..., w_n)$, we'd like to define an embedding vector $Emb_s(s) \in \mathbb{R}^d$ for some $d > 0$. The authors of this paper propose to compute it from the embeddings of words $w_i$, let's call them $Emb_w(w_i)$, so that $...


5

Let's call word2vec vector model $W$ & glove $G$. Now, an embedding is just a vector and $W$ is a vector space. These two embeddings are in different vector spaces. You need to either align the 2 vector spaces like in this paper by Mikolov. The idea is that even though vector spaces are almost isomorphic, they are mostly at an angle and you need to ...


5

You can think of it in terms of physical analogy. You can take a flat surface, like a table, and arrange 30 balls on it. Then you can cut legs from the table and replace it with a single leg. In order to figure out where to put this leg you need to find center of mass of all 30 balls on the table. Assuming that each ball has the same size and weight than ...


4

The Spark documentation for this kind of thing doesn't seem very thorough, so I looked at the source. There's a comment here saying // Need not divide with the norm of the given vector since it is constant. This seems consistent with the following code. So, it seems that findSynonyms doesn't actually return cosine distances, but rather cosine distances ...


4

They both capture the word semantics. Not only W, sometimes W' is also used as word vectors. Even in somecases (W+W')/2 has also been used and better results in that particular task have been obtained. Another thing to notice is that no activation function is used after the hidden layer, so the transformation from input to output is W[i]*W'^T for any ...


4

I'm unclear why this is exactly the case- why are non-negative elements not useful for comparing documents that don't share terms? Just because two documents don't share terms doesn't mean they're dissimilar. In other words, Bag of Words model works good if it says something is similar (of course you can run into problems with similar words with different ...


4

There is no one 'right' way to turn wordvectors back into words. The issue is that the words themselves form a discrete set of points in the embedding space, and so the output of a model is very unlikely to be exactly equal to the location of any word. Typically if your model emits a vector $v$ then interpreting it as a word is done by finding a word $w$ ...


3

Word embeddings mostly help because they can be pre-trained in an unsupervised way on large amounts of text. As a result, NN learns continuous representation of words in a space where words with similar meaning are close to each other. Since NN activation function is continuous (sigmoid or simililar) such representations are inherently easier for NN to work ...


3

There's been some work recently on dynamically assigning word2vec (skip gram) dimension using Boltzmann machines. Check out this paper: "Infinite dimensional word embeddings" -Nalsnick, Ravi The basic idea is to let your training set dictate the dimensionality of your word2vec model, which is penalized by a regularization term that's related to the ...


Only top voted, non community-wiki answers of a minimum length are eligible