I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino acids, so the vocabulary size is only around 20.
I played with different data sizes (tens of millions of peptides), window sizes, and embedding dimensions. Out of curiosity, I tried an embedding size of two. To my surprise, the output vectors lie in a single line, there is no PCA or anything since each embedding is only 2 dimensional, so a simple scatter plot is sufficient for visualization.
I suppose there is some theoretical reason to explain why this is the case reading how skip-gram worked, http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/. Although I was using CBOW here, but I thought it is basically the same to skip-gram except that context and center word are reversed.
Here is my thought, looking at the network from the above post,
In the context of amino acids,
- the input would be 20-dimension one-hot encoded vectors
- there are only two hidden neurons without any non-linear transformation
- the output is also 20-dimension one-hot encoded vectors.
While writing this question, I speculate that 3D embeddings might be on a plane and confirmed it:
=> rotate =>
If you compare the relative positions of the amino acids in the 3D case with those in the 2D case, they're quite consistent.
I expect this pattern to generalize. To phrase it, suppose the embedding dimension is k, then k-D embeddings will lie in a (k-1)-D-dimension hyperplane.
Question: But I haven't figured out a good explanation for it yet, could anyone shed some light on this, please?
I modified the command to use an embedding size of 2 for the text8 dataset that is downloaded and used by the program for demonstration purpose. (If you'd like to try yourself, download code from https://github.com/dav/word2vec, and modify the
$BIN_DIR/word2vec \ -train $TEXT_DATA \ -output $VECTOR_DATA \ -cbow 1 \ -size 2 \ # embedding size = 2 -window 8 \ -negative 25 \ -hs 0 \ -sample 1e\ -4 \ -threads 100 \ -binary 0 \ -iter 15
The result is like below (I arbitrarily labelled some of the data points)
Well, this is not as linear as the first graph I showed, but still surprisingly sort of linear. At this time, I tend to think my result on amino acid at the 2D case may not be an artifact.
Continued playing with text8, setting embedding size to 3, and they appear to be mostly on a plane, consistent with my amino acid results.
Per @Alex R.'s request, not sure what it means to calculate tf-idf out of peptides. In this case, I just treated all peptides as a single document, then tf-idf feels basically like term frequency. I calculated it with a function from scikit-learn
from sklearn.feature_extraction.text import TfidfVectorizer vectorizer = TfidfVectorizer( input='filename', lowercase=False, tokenizer=lambda x: x.split(), vocabulary=list(string.ascii_uppercase) ) res = vectorizer.fit_transform([data_file_path'])
As seen, the distribution is not random at all. For those (e.g. B J O U X Z) with low values, either the corresponding amino acid doesn't exist or they represent some sort of ambiguous amino acid. See ambiguous codes here.
Comparing the word2vec embedding results at low dimensions (2D or 3D) for text8 and amino acids, they are consistent. Maybe that's just the way it is, things turn this way somehow, though it would be nice get some intuition, maybe related to high-dimension geometry, which I have little background in that. Here is a quote from another thread, not very informative, but sort of as a clue
If you pick lesser number of dimensions, you will start to lose properties of high dimensional spaces.