# Why word embeddings learned from word2vec are linearly correlated

I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino acids, so the vocabulary size is only around 20.

I played with different data sizes (tens of millions of peptides), window sizes, and embedding dimensions. Out of curiosity, I tried an embedding size of two. To my surprise, the output vectors lie in a single line, there is no PCA or anything since each embedding is only 2 dimensional, so a simple scatter plot is sufficient for visualization.

I suppose there is some theoretical reason to explain why this is the case reading how skip-gram worked, http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/. Although I was using CBOW here, but I thought it is basically the same to skip-gram except that context and center word are reversed.

Here is my thought, looking at the network from the above post,

In the context of amino acids,

• the input would be 20-dimension one-hot encoded vectors
• there are only two hidden neurons without any non-linear transformation
• the output is also 20-dimension one-hot encoded vectors.

While writing this question, I speculate that 3D embeddings might be on a plane and confirmed it:

=> rotate =>

If you compare the relative positions of the amino acids in the 3D case with those in the 2D case, they're quite consistent.

I expect this pattern to generalize. To phrase it, suppose the embedding dimension is k, then k-D embeddings will lie in a (k-1)-D-dimension hyperplane.

Question: But I haven't figured out a good explanation for it yet, could anyone shed some light on this, please?

Update (2018-03-28):

I modified the command to use an embedding size of 2 for the text8 dataset that is downloaded and used by the program for demonstration purpose. (If you'd like to try yourself, download code from https://github.com/dav/word2vec, and modify the word2vec/scripts/create-text8-vector-data.sh file.)

$BIN_DIR/word2vec \ -train$TEXT_DATA \
-output $VECTOR_DATA \ -cbow 1 \ -size 2 \ # embedding size = 2 -window 8 \ -negative 25 \ -hs 0 \ -sample 1e\ -4 \ -threads 100 \ -binary 0 \ -iter 15  The result is like below (I arbitrarily labelled some of the data points) Well, this is not as linear as the first graph I showed, but still surprisingly sort of linear. At this time, I tend to think my result on amino acid at the 2D case may not be an artifact. Update (2018-03-29): Continued playing with text8, setting embedding size to 3, and they appear to be mostly on a plane, consistent with my amino acid results. tf-idf calculated based on 10 million peptide sequences Per @Alex R.'s request, not sure what it means to calculate tf-idf out of peptides. In this case, I just treated all peptides as a single document, then tf-idf feels basically like term frequency. I calculated it with a function from scikit-learn from sklearn.feature_extraction.text import TfidfVectorizer vectorizer = TfidfVectorizer( input='filename', lowercase=False, tokenizer=lambda x: x.split(), vocabulary=list(string.ascii_uppercase) ) res = vectorizer.fit_transform([data_file_path'])  As seen, the distribution is not random at all. For those (e.g. B J O U X Z) with low values, either the corresponding amino acid doesn't exist or they represent some sort of ambiguous amino acid. See ambiguous codes here. Stage summary Comparing the word2vec embedding results at low dimensions (2D or 3D) for text8 and amino acids, they are consistent. Maybe that's just the way it is, things turn this way somehow, though it would be nice get some intuition, maybe related to high-dimension geometry, which I have little background in that. Here is a quote from another thread, not very informative, but sort of as a clue If you pick lesser number of dimensions, you will start to lose properties of high dimensional spaces. • Out of curiosity, could you post PCA results as well? Also what's your window size? – Alex R. Mar 27 '18 at 17:14 • These are from window size=5. Do you mean PCA for the 3D embeddings? That would be very similar to the first 3D plot since I intentionally chose a viewpoint so that all vectors look like on on a 2D plane. Indeed, they are on a 2D plane embedded in a 3D space, following the pattern I described above, I just haven't figured out why yet. It's a very interesting observation, I thought. – zyxue Mar 27 '18 at 17:25 • This seems highly suspicious. Are you sure your data has been processed correctly? As a sanity check if you look at the TF-IDF matrix of co-occuring word frequencies, does it look right? – Alex R. Mar 27 '18 at 19:37 • @Alex R. My input is just a text file of peptides, one peptide per line. (e.g. M A Q Q R R G G F K R R K K V D F I A A N K I E V V D Y K D T E L L K R F I S E R G K I L P R R V T G T S A K N Q R K V V N A I K R A R V M A L L P F V A E D Q N), also the command I used is $BIN_DIR/word2vec -train ${input} -output${output} -cbow 1 -size ${size} -window${window_size} -hs 0 -threads 128. Is there anything obviously wrong? – zyxue Mar 27 '18 at 19:58
• I don't have extensive experience with NLP, with TF-IDF, what do we expect it look light? In this case, do you mean treating each peptide as a document? – zyxue Mar 27 '18 at 20:08